2021-11-20T05:15:32.109 INFO:root:teuthology version: 1.1.0-6cfddd5e 2021-11-20T05:15:32.111 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} --name yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514741 -- /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514741/orig.config.yaml 2021-11-20T05:15:32.187 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-11-20T05:15:32.287 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514741 branch: wip-yuri2-testing-2021-11-19-1714-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} email: ceph-qa@ceph.io first_in_suite: false job_id: '6514741' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri2-testing-2021-11-19-1714-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 15m rados osd op timeout: 15m global: mon pg warn min per osd: 0 mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri2-testing-2021-11-19-1714-pacific sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa owner: scheduled_yuriw@teuthology priority: 51 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri2-testing-2021-11-19-1714-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa targets: smithi058.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDD/x8BhPThXEUjdnwppoxyS9N9GZygW4lJsBgLaQaO0v4Stn73jyEgVqgKC0iexkosf7R1fL79Aay4dlbtPDSHm8o3k1j2oFBPp/2QvKYeYyhZnCJ9MwSbvGqPejPmDz349uw9rtGs+SmngXwgAIoADBQi2n/d6zXCduFW3VP7HcQoAKzFPlobU/SvtQoy+okai5WYfM15rL1RQkclKZquvgqvYpfVM0avxC9IUgbAs9izDz6GAq9u/QJGDaIoV0NHxu2F5BYoGAgIb3tsxSGZiZ69sD4UKGwmIAf/N9pXQzFBRv3X5aBlWidWs9gH9KzVMwc3XHeyxygK8w3BDj58RJ71F3yIo8lkh0q3xvAx8m20KdkcmqGcscgu06xal9XYMLJKd9JZYR04GtbIjT6V0FIJ9Gn/DNkPij21KY5gSQCIEBkrWd9bXFhAmueSmgthOlRHMaumhZGfAsflZhFJm4EBbb0ZjvkFIbdVYDATjHSMQcKVZai0bJlSBqxZ/9k= smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCihhPaVlU52OnfFyY9S6+9w6DTxBx5EWwFWFTyqIWopH8vhsgwkX9OY1EltQQUYYjYI+QgfB/PyXlZOydWVsZJw0kodbK+tJtL54GvUPiLLCo5MtrYrVn08VowY1Plk3CTT1pN3aJjVmlhF2Mh7ws/o7ToKgFlZffvR4QE/wKC9o2KMm+l9DAQ4hXI0xJgbmXXZi1jVPCp4bViCT7UxOhvCNcB8t2bqrUyVLq5HocSa90AS/JD6vLBA2LIY7v+ZQ2qwtHiDW6MIoNtAJ9e1gwIKM9eFhYoyB57oQb2lb7cG5Fg7rywPBoeojeAig5K03tqmtKvMeax4A3H79lt06zFe6U5b3G3mIoNNoysTRRrF47NCXxII9s1YgooTha9oFO0piXClyy341ALchY40pbIZRfBq0moTV3GzblYo/CPO3u2/pTDgwlase2zcbQHxfugTUfU32Z6tiLq+33oFPiIuzEYx1ICfpC8ZC4YFasBdwdXLZ7PJ4hOzJDi8txc/Gc= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: exclude_packages: - ceph-volume tag: v16.2.4 - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - ceph --format=json mds versions | jq -e ". | add == 4" - fs.pre_upgrade_save: null - kclient: null - print: '**** done kclient client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology_branch: master teuthology_sha1: 6cfddd5e1c7d5ede1367cf3087c21af44eb88356 timestamp: 2021-11-20_04:32:48 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; ceph fs dump; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.20261 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2021-11-20T05:15:32.288 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa; will attempt to use it 2021-11-20T05:15:32.290 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks 2021-11-20T05:15:32.290 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-11-20T05:15:32.295 INFO:teuthology.task.internal:Checking packages... 2021-11-20T05:15:32.321 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa' 2021-11-20T05:15:32.322 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-11-20T05:15:32.322 INFO:teuthology.packaging:ref: None 2021-11-20T05:15:32.323 INFO:teuthology.packaging:tag: None 2021-11-20T05:15:32.323 INFO:teuthology.packaging:branch: wip-yuri2-testing-2021-11-19-1714-pacific 2021-11-20T05:15:32.323 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T05:15:32.324 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri2-testing-2021-11-19-1714-pacific 2021-11-20T05:15:32.561 INFO:teuthology.task.internal:Found packages for ceph version 16.2.6-633.g9a97ad1b 2021-11-20T05:15:32.562 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-11-20T05:15:32.571 INFO:teuthology.task.internal:no buildpackages task found 2021-11-20T05:15:32.572 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-11-20T05:15:32.609 INFO:teuthology.task.internal:Saving configuration 2021-11-20T05:15:32.642 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-11-20T05:15:32.653 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-20T05:15:32.680 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi058.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514741', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2021-11-20 05:10:04.956645', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCEbGAYa2yH8NrxIJjlN8e83YxvQD8T8t+tWfeDcDoZkvnfEIGvc/KJpAE48Qa24DMy/pNJbFqZSjEcRsoXo1Ow='} 2021-11-20T05:15:32.706 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514741', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2021-11-20 05:10:04.955287', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFNvCD0LnOuee7NqXxHZKdmfuaAqJGPirQCC6sxlWL1UYbJurTkG+qKli8GmPKN2ru2B2xQejNpAfBnoFlSOQew='} 2021-11-20T05:15:32.706 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-11-20T05:15:32.730 INFO:teuthology.task.internal:roles: ubuntu@smithi058.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2021-11-20T05:15:32.731 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2021-11-20T05:15:32.731 INFO:teuthology.run_tasks:Running task console_log... 2021-11-20T05:15:32.883 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff4b61f4598>, signals=[15]) 2021-11-20T05:15:32.885 INFO:teuthology.run_tasks:Running task internal.connect... 2021-11-20T05:15:32.909 INFO:teuthology.task.internal:Opening connections... 2021-11-20T05:15:32.911 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi058.front.sepia.ceph.com 2021-11-20T05:15:32.920 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi058.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T05:15:33.037 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi187.front.sepia.ceph.com 2021-11-20T05:15:33.038 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T05:15:33.112 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-11-20T05:15:33.121 DEBUG:teuthology.orchestra.run.smithi058:> uname -m 2021-11-20T05:15:33.140 INFO:teuthology.orchestra.run.smithi058.stdout:x86_64 2021-11-20T05:15:33.141 DEBUG:teuthology.orchestra.run.smithi058:> cat /etc/os-release 2021-11-20T05:15:33.198 INFO:teuthology.orchestra.run.smithi058.stdout:NAME="CentOS Stream" 2021-11-20T05:15:33.199 INFO:teuthology.orchestra.run.smithi058.stdout:VERSION="8" 2021-11-20T05:15:33.199 INFO:teuthology.orchestra.run.smithi058.stdout:ID="centos" 2021-11-20T05:15:33.199 INFO:teuthology.orchestra.run.smithi058.stdout:ID_LIKE="rhel fedora" 2021-11-20T05:15:33.200 INFO:teuthology.orchestra.run.smithi058.stdout:VERSION_ID="8" 2021-11-20T05:15:33.200 INFO:teuthology.orchestra.run.smithi058.stdout:PLATFORM_ID="platform:el8" 2021-11-20T05:15:33.201 INFO:teuthology.orchestra.run.smithi058.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-20T05:15:33.202 INFO:teuthology.orchestra.run.smithi058.stdout:ANSI_COLOR="0;31" 2021-11-20T05:15:33.202 INFO:teuthology.orchestra.run.smithi058.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-20T05:15:33.203 INFO:teuthology.orchestra.run.smithi058.stdout:HOME_URL="https://centos.org/" 2021-11-20T05:15:33.203 INFO:teuthology.orchestra.run.smithi058.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-20T05:15:33.204 INFO:teuthology.orchestra.run.smithi058.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-20T05:15:33.204 INFO:teuthology.orchestra.run.smithi058.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-20T05:15:33.206 INFO:teuthology.lock.ops:Updating smithi058.front.sepia.ceph.com on lock server 2021-11-20T05:15:33.227 DEBUG:teuthology.orchestra.run.smithi187:> uname -m 2021-11-20T05:15:33.249 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2021-11-20T05:15:33.249 DEBUG:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2021-11-20T05:15:33.306 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Stream" 2021-11-20T05:15:33.307 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="8" 2021-11-20T05:15:33.307 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos" 2021-11-20T05:15:33.307 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora" 2021-11-20T05:15:33.308 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="8" 2021-11-20T05:15:33.308 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el8" 2021-11-20T05:15:33.308 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-20T05:15:33.308 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2021-11-20T05:15:33.308 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-20T05:15:33.309 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://centos.org/" 2021-11-20T05:15:33.309 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-20T05:15:33.309 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-20T05:15:33.309 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-20T05:15:33.310 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2021-11-20T05:15:33.327 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-11-20T05:15:33.344 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-11-20T05:15:33.377 INFO:teuthology.task.internal:Checking for old test directory... 2021-11-20T05:15:33.378 DEBUG:teuthology.orchestra.run.smithi058:> test '!' -e /home/ubuntu/cephtest 2021-11-20T05:15:33.384 DEBUG:teuthology.orchestra.run.smithi187:> test '!' -e /home/ubuntu/cephtest 2021-11-20T05:15:33.401 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-11-20T05:15:33.412 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-11-20T05:15:33.413 DEBUG:teuthology.orchestra.run.smithi058:> test -z $(ls -A /var/lib/ceph) 2021-11-20T05:15:33.441 DEBUG:teuthology.orchestra.run.smithi187:> test -z $(ls -A /var/lib/ceph) 2021-11-20T05:15:33.481 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-11-20T05:15:33.540 INFO:teuthology.run_tasks:Running task kernel... 2021-11-20T05:15:33.562 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2021-11-20T05:15:33.563 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-11-20T05:15:33.564 DEBUG:teuthology.orchestra.run.smithi058:> uname -r 2021-11-20T05:15:33.565 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2021-11-20T05:15:33.583 INFO:teuthology.orchestra.run.smithi058.stdout:4.18.0-301.1.el8.x86_64 2021-11-20T05:15:33.583 INFO:teuthology.task.kernel:Running kernel on smithi058: 4.18.0-301.1.el8.x86_64 2021-11-20T05:15:33.584 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum install -y kernel 2021-11-20T05:15:33.586 INFO:teuthology.orchestra.run.smithi187.stdout:4.18.0-301.1.el8.x86_64 2021-11-20T05:15:33.587 INFO:teuthology.task.kernel:Running kernel on smithi187: 4.18.0-301.1.el8.x86_64 2021-11-20T05:15:33.588 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum install -y kernel 2021-11-20T05:15:36.376 INFO:teuthology.orchestra.run.smithi058.stdout:CentOS-8 - AppStream 35 MB/s | 17 MB 00:00 2021-11-20T05:15:36.464 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-8 - AppStream 37 MB/s | 17 MB 00:00 2021-11-20T05:15:39.575 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-8 - Base 60 MB/s | 15 MB 00:00 2021-11-20T05:15:39.774 INFO:teuthology.orchestra.run.smithi058.stdout:CentOS-8 - Base 53 MB/s | 15 MB 00:00 2021-11-20T05:15:41.770 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-8 - Extras 244 kB/s | 16 kB 00:00 2021-11-20T05:15:42.259 INFO:teuthology.orchestra.run.smithi058.stdout:CentOS-8 - Extras 228 kB/s | 16 kB 00:00 2021-11-20T05:15:42.618 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 14 MB/s | 11 MB 00:00 2021-11-20T05:15:43.201 INFO:teuthology.orchestra.run.smithi058.stdout:Extra Packages for Enterprise Linux 13 MB/s | 11 MB 00:00 2021-11-20T05:15:44.991 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 3.8 MB/s | 24 kB 00:00 2021-11-20T05:15:45.716 INFO:teuthology.orchestra.run.smithi058.stdout:lab-extras 4.1 MB/s | 24 kB 00:00 2021-11-20T05:15:46.942 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-4.18.0-240.22.1.el8_3.x86_64 is already installed. 2021-11-20T05:15:46.943 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-4.18.0-301.1.el8.x86_64 is already installed. 2021-11-20T05:15:46.943 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-11-20T05:15:47.258 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:15:47.258 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:15:47.259 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2021-11-20T05:15:47.259 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:15:47.259 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2021-11-20T05:15:47.260 INFO:teuthology.orchestra.run.smithi187.stdout: kernel x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 7.0 M 2021-11-20T05:15:47.260 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-core x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 38 M 2021-11-20T05:15:47.260 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 30 M 2021-11-20T05:15:47.260 INFO:teuthology.orchestra.run.smithi187.stdout:Removing: 2021-11-20T05:15:47.261 INFO:teuthology.orchestra.run.smithi187.stdout: kernel x86_64 4.18.0-240.el8 @anaconda 0 2021-11-20T05:15:47.261 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-core x86_64 4.18.0-240.el8 @anaconda 62 M 2021-11-20T05:15:47.261 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules x86_64 4.18.0-240.el8 @anaconda 21 M 2021-11-20T05:15:47.261 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:15:47.262 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:15:47.262 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:15:47.262 INFO:teuthology.orchestra.run.smithi187.stdout:Install 3 Packages 2021-11-20T05:15:47.263 INFO:teuthology.orchestra.run.smithi187.stdout:Remove 3 Packages 2021-11-20T05:15:47.263 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:15:47.264 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 74 M 2021-11-20T05:15:47.264 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2021-11-20T05:15:47.433 INFO:teuthology.orchestra.run.smithi187.stdout:(1/3): kernel-4.18.0-348.2.1.el8_5.x86_64.rpm 40 MB/s | 7.0 MB 00:00 2021-11-20T05:15:47.846 INFO:teuthology.orchestra.run.smithi058.stdout:Package kernel-4.18.0-240.22.1.el8_3.x86_64 is already installed. 2021-11-20T05:15:47.846 INFO:teuthology.orchestra.run.smithi058.stdout:Package kernel-4.18.0-301.1.el8.x86_64 is already installed. 2021-11-20T05:15:47.846 INFO:teuthology.orchestra.run.smithi058.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-11-20T05:15:48.170 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:15:48.170 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:15:48.170 INFO:teuthology.orchestra.run.smithi058.stdout: Package Arch Version Repository Size 2021-11-20T05:15:48.170 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:15:48.171 INFO:teuthology.orchestra.run.smithi058.stdout:Installing: 2021-11-20T05:15:48.171 INFO:teuthology.orchestra.run.smithi058.stdout: kernel x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 7.0 M 2021-11-20T05:15:48.171 INFO:teuthology.orchestra.run.smithi058.stdout: kernel-core x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 38 M 2021-11-20T05:15:48.171 INFO:teuthology.orchestra.run.smithi058.stdout: kernel-modules x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 30 M 2021-11-20T05:15:48.172 INFO:teuthology.orchestra.run.smithi058.stdout:Removing: 2021-11-20T05:15:48.172 INFO:teuthology.orchestra.run.smithi058.stdout: kernel x86_64 4.18.0-240.el8 @anaconda 0 2021-11-20T05:15:48.172 INFO:teuthology.orchestra.run.smithi058.stdout: kernel-core x86_64 4.18.0-240.el8 @anaconda 62 M 2021-11-20T05:15:48.172 INFO:teuthology.orchestra.run.smithi058.stdout: kernel-modules x86_64 4.18.0-240.el8 @anaconda 21 M 2021-11-20T05:15:48.173 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:15:48.173 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:15:48.173 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:15:48.173 INFO:teuthology.orchestra.run.smithi058.stdout:Install 3 Packages 2021-11-20T05:15:48.173 INFO:teuthology.orchestra.run.smithi058.stdout:Remove 3 Packages 2021-11-20T05:15:48.174 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:15:48.174 INFO:teuthology.orchestra.run.smithi058.stdout:Total download size: 74 M 2021-11-20T05:15:48.174 INFO:teuthology.orchestra.run.smithi058.stdout:Downloading Packages: 2021-11-20T05:15:48.230 INFO:teuthology.orchestra.run.smithi187.stdout:(2/3): kernel-modules-4.18.0-348.2.1.el8_5.x86_ 31 MB/s | 30 MB 00:00 2021-11-20T05:15:48.492 INFO:teuthology.orchestra.run.smithi058.stdout:(1/3): kernel-4.18.0-348.2.1.el8_5.x86_64.rpm 22 MB/s | 7.0 MB 00:00 2021-11-20T05:15:48.535 INFO:teuthology.orchestra.run.smithi187.stdout:(3/3): kernel-core-4.18.0-348.2.1.el8_5.x86_64. 29 MB/s | 38 MB 00:01 2021-11-20T05:15:48.543 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:15:48.544 INFO:teuthology.orchestra.run.smithi187.stdout:Total 58 MB/s | 74 MB 00:01 2021-11-20T05:15:48.671 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2021-11-20T05:15:49.131 INFO:teuthology.orchestra.run.smithi058.stdout:(2/3): kernel-core-4.18.0-348.2.1.el8_5.x86_64. 39 MB/s | 38 MB 00:00 2021-11-20T05:15:49.309 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2021-11-20T05:15:49.309 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2021-11-20T05:15:49.575 INFO:teuthology.orchestra.run.smithi058.stdout:(3/3): kernel-modules-4.18.0-348.2.1.el8_5.x86_ 21 MB/s | 30 MB 00:01 2021-11-20T05:15:49.578 INFO:teuthology.orchestra.run.smithi058.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:15:49.580 INFO:teuthology.orchestra.run.smithi058.stdout:Total 53 MB/s | 74 MB 00:01 2021-11-20T05:15:49.726 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction check 2021-11-20T05:15:50.436 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2021-11-20T05:15:50.464 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2021-11-20T05:15:50.567 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction check succeeded. 2021-11-20T05:15:50.567 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction test 2021-11-20T05:15:51.668 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction test succeeded. 2021-11-20T05:15:51.672 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction 2021-11-20T05:15:53.115 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2021-11-20T05:15:53.620 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T05:15:54.502 INFO:teuthology.orchestra.run.smithi058.stdout: Preparing : 1/1 2021-11-20T05:15:55.135 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T05:15:57.776 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T05:15:58.153 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T05:15:59.503 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T05:15:59.857 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T05:16:01.213 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T05:16:01.405 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-20T05:16:01.405 INFO:teuthology.orchestra.run.smithi187.stdout: Erasing : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T05:16:01.627 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T05:16:01.627 INFO:teuthology.orchestra.run.smithi187.stdout: Erasing : kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-20T05:16:03.150 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T05:16:03.345 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : kernel-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-20T05:16:03.346 INFO:teuthology.orchestra.run.smithi058.stdout: Erasing : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T05:16:03.545 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T05:16:03.545 INFO:teuthology.orchestra.run.smithi058.stdout: Erasing : kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-20T05:16:03.741 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-20T05:16:03.937 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:16:03.948 INFO:teuthology.orchestra.run.smithi187.stdout: Erasing : kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:16:05.643 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-20T05:16:05.854 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:16:05.863 INFO:teuthology.orchestra.run.smithi058.stdout: Erasing : kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:16:06.802 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:16:08.911 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:16:27.705 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 6/6 2021-11-20T05:16:28.112 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:16:28.112 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T05:16:28.113 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-core-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T05:16:28.113 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-20T05:16:28.113 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T05:16:28.113 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-core-4.18.0-240.el8.x86_64 5/6 2021-11-20T05:16:28.562 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-modules-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:16:28.562 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:16:28.562 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2021-11-20T05:16:28.563 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:16:28.563 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-core-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:16:28.563 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:16:28.563 INFO:teuthology.orchestra.run.smithi187.stdout:Removed: 2021-11-20T05:16:28.564 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-4.18.0-240.el8.x86_64 kernel-core-4.18.0-240.el8.x86_64 2021-11-20T05:16:28.564 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-4.18.0-240.el8.x86_64 2021-11-20T05:16:28.564 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:16:28.564 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:16:28.702 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel --last | head -n 1 2021-11-20T05:16:28.868 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Sat 20 Nov 2021 05:16:01 AM UTC 2021-11-20T05:16:28.870 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:16:28.870 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-301.1.el8.x86_64 Expected: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:16:28.870 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-11-20T05:16:28.871 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-11-20T05:16:28.871 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2021-11-20T05:16:28.871 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi187.front.sepia.ceph.com, path=None, version=distro) 2021-11-20T05:16:28.872 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel --last | head -n 1 2021-11-20T05:16:29.014 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Sat 20 Nov 2021 05:16:01 AM UTC 2021-11-20T05:16:29.016 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:16:29.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo rpm -qi grub2-tools 2021-11-20T05:16:29.048 INFO:teuthology.orchestra.run.smithi187.stdout:Name : grub2-tools 2021-11-20T05:16:29.048 INFO:teuthology.orchestra.run.smithi187.stdout:Epoch : 1 2021-11-20T05:16:29.048 INFO:teuthology.orchestra.run.smithi187.stdout:Version : 2.02 2021-11-20T05:16:29.049 INFO:teuthology.orchestra.run.smithi187.stdout:Release : 99.el8 2021-11-20T05:16:29.049 INFO:teuthology.orchestra.run.smithi187.stdout:Architecture: x86_64 2021-11-20T05:16:29.049 INFO:teuthology.orchestra.run.smithi187.stdout:Install Date: Thu 13 May 2021 06:08:01 PM UTC 2021-11-20T05:16:29.049 INFO:teuthology.orchestra.run.smithi187.stdout:Group : System Environment/Base 2021-11-20T05:16:29.050 INFO:teuthology.orchestra.run.smithi187.stdout:Size : 9264704 2021-11-20T05:16:29.050 INFO:teuthology.orchestra.run.smithi187.stdout:License : GPLv3+ 2021-11-20T05:16:29.050 INFO:teuthology.orchestra.run.smithi187.stdout:Signature : RSA/SHA256, Tue 02 Mar 2021 09:11:55 PM UTC, Key ID 05b555b38483c65d 2021-11-20T05:16:29.050 INFO:teuthology.orchestra.run.smithi187.stdout:Source RPM : grub2-2.02-99.el8.src.rpm 2021-11-20T05:16:29.050 INFO:teuthology.orchestra.run.smithi187.stdout:Build Date : Tue 02 Mar 2021 09:03:27 PM UTC 2021-11-20T05:16:29.051 INFO:teuthology.orchestra.run.smithi187.stdout:Build Host : kbuilder.bsys.centos.org 2021-11-20T05:16:29.051 INFO:teuthology.orchestra.run.smithi187.stdout:Relocations : (not relocatable) 2021-11-20T05:16:29.051 INFO:teuthology.orchestra.run.smithi187.stdout:Packager : CentOS BuildSystem 2021-11-20T05:16:29.051 INFO:teuthology.orchestra.run.smithi187.stdout:Vendor : CentOS 2021-11-20T05:16:29.052 INFO:teuthology.orchestra.run.smithi187.stdout:URL : http://www.gnu.org/software/grub/ 2021-11-20T05:16:29.052 INFO:teuthology.orchestra.run.smithi187.stdout:Summary : Support tools for GRUB. 2021-11-20T05:16:29.052 INFO:teuthology.orchestra.run.smithi187.stdout:Description : 2021-11-20T05:16:29.052 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:16:29.052 INFO:teuthology.orchestra.run.smithi187.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-11-20T05:16:29.053 INFO:teuthology.orchestra.run.smithi187.stdout:customizable bootloader with modular architecture. It supports a rich 2021-11-20T05:16:29.053 INFO:teuthology.orchestra.run.smithi187.stdout:variety of kernel formats, file systems, computer architectures and 2021-11-20T05:16:29.053 INFO:teuthology.orchestra.run.smithi187.stdout:hardware devices. 2021-11-20T05:16:29.053 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:16:29.054 INFO:teuthology.orchestra.run.smithi187.stdout:This subpackage provides tools for support of all platforms. 2021-11-20T05:16:29.055 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-11-20T05:16:29.055 INFO:teuthology.task.kernel:Updating grub on smithi187 to boot 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:16:29.055 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-11-20T05:16:30.469 INFO:teuthology.orchestra.run.smithi187.stderr:Generating grub configuration file ... 2021-11-20T05:16:31.134 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 6/6 2021-11-20T05:16:31.506 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:16:31.507 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : kernel-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T05:16:31.507 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : kernel-core-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T05:16:31.508 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-20T05:16:31.508 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T05:16:31.509 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : kernel-core-4.18.0-240.el8.x86_64 5/6 2021-11-20T05:16:32.034 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : kernel-modules-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:16:32.034 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:16:32.034 INFO:teuthology.orchestra.run.smithi058.stdout:Installed: 2021-11-20T05:16:32.035 INFO:teuthology.orchestra.run.smithi058.stdout: kernel-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:16:32.035 INFO:teuthology.orchestra.run.smithi058.stdout: kernel-core-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:16:32.035 INFO:teuthology.orchestra.run.smithi058.stdout: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:16:32.035 INFO:teuthology.orchestra.run.smithi058.stdout:Removed: 2021-11-20T05:16:32.036 INFO:teuthology.orchestra.run.smithi058.stdout: kernel-4.18.0-240.el8.x86_64 kernel-core-4.18.0-240.el8.x86_64 2021-11-20T05:16:32.036 INFO:teuthology.orchestra.run.smithi058.stdout: kernel-modules-4.18.0-240.el8.x86_64 2021-11-20T05:16:32.036 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:16:32.036 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:16:32.165 DEBUG:teuthology.orchestra.run.smithi058:> rpm -q kernel --last | head -n 1 2021-11-20T05:16:32.329 INFO:teuthology.orchestra.run.smithi058.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Sat 20 Nov 2021 05:16:03 AM UTC 2021-11-20T05:16:32.331 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:16:32.331 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-301.1.el8.x86_64 Expected: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:16:32.331 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-11-20T05:16:32.331 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-11-20T05:16:32.332 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2021-11-20T05:16:32.332 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi058.front.sepia.ceph.com, path=None, version=distro) 2021-11-20T05:16:32.332 DEBUG:teuthology.orchestra.run.smithi058:> rpm -q kernel --last | head -n 1 2021-11-20T05:16:32.409 INFO:teuthology.orchestra.run.smithi187.stderr:done 2021-11-20T05:16:32.411 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2021-11-20T05:16:32.447 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.6PwZPaRdt2 2021-11-20T05:16:32.447 DEBUG:teuthology.orchestra.run.smithi187:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6PwZPaRdt2 2021-11-20T05:16:32.481 INFO:teuthology.orchestra.run.smithi058.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Sat 20 Nov 2021 05:16:03 AM UTC 2021-11-20T05:16:32.483 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:16:32.483 DEBUG:teuthology.orchestra.run.smithi058:> sudo rpm -qi grub2-tools 2021-11-20T05:16:32.515 INFO:teuthology.orchestra.run.smithi058.stdout:Name : grub2-tools 2021-11-20T05:16:32.516 INFO:teuthology.orchestra.run.smithi058.stdout:Epoch : 1 2021-11-20T05:16:32.516 INFO:teuthology.orchestra.run.smithi058.stdout:Version : 2.02 2021-11-20T05:16:32.516 INFO:teuthology.orchestra.run.smithi058.stdout:Release : 99.el8 2021-11-20T05:16:32.517 INFO:teuthology.orchestra.run.smithi058.stdout:Architecture: x86_64 2021-11-20T05:16:32.517 INFO:teuthology.orchestra.run.smithi058.stdout:Install Date: Thu 13 May 2021 06:08:01 PM UTC 2021-11-20T05:16:32.517 INFO:teuthology.orchestra.run.smithi058.stdout:Group : System Environment/Base 2021-11-20T05:16:32.517 INFO:teuthology.orchestra.run.smithi058.stdout:Size : 9264704 2021-11-20T05:16:32.517 INFO:teuthology.orchestra.run.smithi058.stdout:License : GPLv3+ 2021-11-20T05:16:32.518 INFO:teuthology.orchestra.run.smithi058.stdout:Signature : RSA/SHA256, Tue 02 Mar 2021 09:11:55 PM UTC, Key ID 05b555b38483c65d 2021-11-20T05:16:32.518 INFO:teuthology.orchestra.run.smithi058.stdout:Source RPM : grub2-2.02-99.el8.src.rpm 2021-11-20T05:16:32.518 INFO:teuthology.orchestra.run.smithi058.stdout:Build Date : Tue 02 Mar 2021 09:03:27 PM UTC 2021-11-20T05:16:32.518 INFO:teuthology.orchestra.run.smithi058.stdout:Build Host : kbuilder.bsys.centos.org 2021-11-20T05:16:32.519 INFO:teuthology.orchestra.run.smithi058.stdout:Relocations : (not relocatable) 2021-11-20T05:16:32.519 INFO:teuthology.orchestra.run.smithi058.stdout:Packager : CentOS BuildSystem 2021-11-20T05:16:32.519 INFO:teuthology.orchestra.run.smithi058.stdout:Vendor : CentOS 2021-11-20T05:16:32.519 INFO:teuthology.orchestra.run.smithi058.stdout:URL : http://www.gnu.org/software/grub/ 2021-11-20T05:16:32.520 INFO:teuthology.orchestra.run.smithi058.stdout:Summary : Support tools for GRUB. 2021-11-20T05:16:32.520 INFO:teuthology.orchestra.run.smithi058.stdout:Description : 2021-11-20T05:16:32.521 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:16:32.521 INFO:teuthology.orchestra.run.smithi058.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-11-20T05:16:32.521 INFO:teuthology.orchestra.run.smithi058.stdout:customizable bootloader with modular architecture. It supports a rich 2021-11-20T05:16:32.521 INFO:teuthology.orchestra.run.smithi058.stdout:variety of kernel formats, file systems, computer architectures and 2021-11-20T05:16:32.521 INFO:teuthology.orchestra.run.smithi058.stdout:hardware devices. 2021-11-20T05:16:32.522 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:16:32.522 INFO:teuthology.orchestra.run.smithi058.stdout:This subpackage provides tools for support of all platforms. 2021-11-20T05:16:32.524 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.6PwZPaRdt2 2021-11-20T05:16:32.525 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-11-20T05:16:32.525 INFO:teuthology.task.kernel:Updating grub on smithi058 to boot 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:16:32.525 DEBUG:teuthology.orchestra.run.smithi058:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-11-20T05:16:32.664 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.6PwZPaRdt2 is 6KB 2021-11-20T05:16:32.715 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.6PwZPaRdt2 2021-11-20T05:16:32.731 DEBUG:teuthology.orchestra.run.smithi187:> sudo /bin/ls /boot/loader/entries || true 2021-11-20T05:16:32.801 INFO:teuthology.orchestra.run.smithi187.stdout:3331cc438d4e47d3b31025a6d3bd39f2-0-rescue.conf 2021-11-20T05:16:32.802 INFO:teuthology.orchestra.run.smithi187.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-240.22.1.el8_3.x86_64.conf 2021-11-20T05:16:32.802 INFO:teuthology.orchestra.run.smithi187.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-301.1.el8.x86_64.conf 2021-11-20T05:16:32.802 INFO:teuthology.orchestra.run.smithi187.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64.conf 2021-11-20T05:16:32.803 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-set-default 3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:16:32.950 DEBUG:teuthology.orchestra.run.smithi187:> sudo shutdown -r now 2021-11-20T05:16:34.060 INFO:teuthology.orchestra.run.smithi058.stderr:Generating grub configuration file ... 2021-11-20T05:16:36.482 INFO:teuthology.orchestra.run.smithi058.stderr:done 2021-11-20T05:16:36.484 DEBUG:teuthology.orchestra.run.smithi058:> mktemp 2021-11-20T05:16:36.534 INFO:teuthology.orchestra.run.smithi058.stdout:/tmp/tmp.OrO6ryaDBy 2021-11-20T05:16:36.535 DEBUG:teuthology.orchestra.run.smithi058:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OrO6ryaDBy 2021-11-20T05:16:36.565 DEBUG:teuthology.orchestra.run.smithi058:> sudo chmod 0666 /tmp/tmp.OrO6ryaDBy 2021-11-20T05:16:36.717 DEBUG:teuthology.orchestra.remote:smithi058:/tmp/tmp.OrO6ryaDBy is 6KB 2021-11-20T05:16:36.768 DEBUG:teuthology.orchestra.run.smithi058:> rm -fr /tmp/tmp.OrO6ryaDBy 2021-11-20T05:16:36.785 DEBUG:teuthology.orchestra.run.smithi058:> sudo /bin/ls /boot/loader/entries || true 2021-11-20T05:16:36.854 INFO:teuthology.orchestra.run.smithi058.stdout:3331cc438d4e47d3b31025a6d3bd39f2-0-rescue.conf 2021-11-20T05:16:36.854 INFO:teuthology.orchestra.run.smithi058.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-240.22.1.el8_3.x86_64.conf 2021-11-20T05:16:36.854 INFO:teuthology.orchestra.run.smithi058.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-301.1.el8.x86_64.conf 2021-11-20T05:16:36.855 INFO:teuthology.orchestra.run.smithi058.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64.conf 2021-11-20T05:16:36.856 DEBUG:teuthology.orchestra.run.smithi058:> sudo grub2-set-default 3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:16:37.001 DEBUG:teuthology.orchestra.run.smithi058:> sudo shutdown -r now 2021-11-20T05:17:02.979 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2021-11-20T05:17:02.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2021-11-20T05:17:02.981 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T05:17:07.008 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2021-11-20T05:17:07.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi058.front.sepia.ceph.com' 2021-11-20T05:17:07.010 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi058.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T05:18:21.538 DEBUG:teuthology.orchestra.remote:timed out 2021-11-20T05:18:25.584 DEBUG:teuthology.orchestra.remote:timed out 2021-11-20T05:18:51.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2021-11-20T05:18:51.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T05:18:51.923 DEBUG:teuthology.orchestra.run.smithi187:> true 2021-11-20T05:18:52.256 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2021-11-20T05:18:52.256 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-348.2.1.el8_5.x86_64"... 2021-11-20T05:18:52.257 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2021-11-20T05:18:52.274 INFO:teuthology.orchestra.run.smithi187.stdout:4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:18:52.275 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-348.2.1.el8_5.x86_64 vs 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:18:52.275 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-11-20T05:18:52.275 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-11-20T05:18:53.276 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2021-11-20T05:18:53.277 DEBUG:teuthology.orchestra.run.smithi187:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-11-20T05:18:53.356 INFO:teuthology.orchestra.run.smithi187.stdout:ttyS1 2021-11-20T05:18:53.371 DEBUG:teuthology.parallel:result is None 2021-11-20T05:18:55.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi058.front.sepia.ceph.com' 2021-11-20T05:18:55.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi058.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T05:18:55.887 DEBUG:teuthology.orchestra.run.smithi058:> true 2021-11-20T05:18:56.158 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi058.front.sepia.ceph.com' 2021-11-20T05:18:56.158 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-348.2.1.el8_5.x86_64"... 2021-11-20T05:18:56.159 DEBUG:teuthology.orchestra.run.smithi058:> uname -r 2021-11-20T05:18:56.175 INFO:teuthology.orchestra.run.smithi058.stdout:4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:18:56.176 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-348.2.1.el8_5.x86_64 vs 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:18:56.176 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-11-20T05:18:56.176 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-11-20T05:18:57.177 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2021-11-20T05:18:57.178 DEBUG:teuthology.orchestra.run.smithi058:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-11-20T05:18:57.267 INFO:teuthology.orchestra.run.smithi058.stdout:ttyS1 2021-11-20T05:18:57.283 DEBUG:teuthology.parallel:result is None 2021-11-20T05:18:57.284 INFO:teuthology.run_tasks:Running task internal.base... 2021-11-20T05:18:57.296 INFO:teuthology.task.internal:Creating test directory... 2021-11-20T05:18:57.297 DEBUG:teuthology.orchestra.run.smithi058:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-11-20T05:18:57.300 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-11-20T05:18:57.318 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-11-20T05:18:57.330 INFO:teuthology.run_tasks:Running task internal.archive... 2021-11-20T05:18:57.354 INFO:teuthology.task.internal:Creating archive directory... 2021-11-20T05:18:57.354 DEBUG:teuthology.orchestra.run.smithi058:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-11-20T05:18:57.358 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-11-20T05:18:57.396 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-11-20T05:18:57.410 INFO:teuthology.task.internal:Enabling coredump saving... 2021-11-20T05:18:57.411 DEBUG:teuthology.orchestra.run.smithi058:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-11-20T05:18:57.421 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-11-20T05:18:57.450 INFO:teuthology.orchestra.run.smithi058.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-20T05:18:57.464 INFO:teuthology.orchestra.run.smithi058.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-20T05:18:57.465 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-20T05:18:57.476 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-20T05:18:57.478 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-11-20T05:18:57.488 INFO:teuthology.task.internal:Configuring sudo... 2021-11-20T05:18:57.489 DEBUG:teuthology.orchestra.run.smithi058:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-11-20T05:18:57.507 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-11-20T05:18:57.546 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-11-20T05:18:57.558 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-11-20T05:18:57.559 DEBUG:teuthology.orchestra.run.smithi058:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-11-20T05:18:57.577 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-11-20T05:18:57.602 DEBUG:teuthology.orchestra.run.smithi058:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-20T05:18:57.677 DEBUG:teuthology.orchestra.run.smithi058:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-20T05:18:57.730 DEBUG:teuthology.orchestra.run.smithi058:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-20T05:18:57.811 DEBUG:teuthology.orchestra.run.smithi058:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-20T05:18:57.857 DEBUG:teuthology.orchestra.run.smithi058:> set -ex 2021-11-20T05:18:57.857 DEBUG:teuthology.orchestra.run.smithi058:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-11-20T05:18:57.924 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-20T05:18:57.969 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-20T05:18:58.023 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-20T05:18:58.116 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-20T05:18:58.155 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2021-11-20T05:18:58.156 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-11-20T05:18:58.223 DEBUG:teuthology.orchestra.run.smithi058:> sudo service rsyslog restart 2021-11-20T05:18:58.226 DEBUG:teuthology.orchestra.run.smithi187:> sudo service rsyslog restart 2021-11-20T05:18:58.270 INFO:teuthology.orchestra.run.smithi058.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-20T05:18:58.318 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-20T05:18:58.731 INFO:teuthology.run_tasks:Running task internal.timer... 2021-11-20T05:18:58.743 INFO:teuthology.task.internal:Starting timer... 2021-11-20T05:18:58.743 INFO:teuthology.run_tasks:Running task pcp... 2021-11-20T05:18:58.774 INFO:teuthology.run_tasks:Running task selinux... 2021-11-20T05:18:58.808 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-11-20T05:18:58.808 DEBUG:teuthology.orchestra.run.smithi058:> sudo service auditd rotate 2021-11-20T05:18:58.866 INFO:teuthology.orchestra.run.smithi058.stdout:Rotating logs: [ OK ] 2021-11-20T05:18:58.867 DEBUG:teuthology.orchestra.run.smithi187:> sudo service auditd rotate 2021-11-20T05:18:58.924 INFO:teuthology.orchestra.run.smithi187.stdout:Rotating logs: [ OK ] 2021-11-20T05:18:58.926 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-11-20T05:18:58.926 DEBUG:teuthology.orchestra.run.smithi058:> /usr/sbin/getenforce 2021-11-20T05:18:58.945 INFO:teuthology.orchestra.run.smithi058.stdout:Permissive 2021-11-20T05:18:58.945 DEBUG:teuthology.orchestra.run.smithi187:> /usr/sbin/getenforce 2021-11-20T05:18:58.976 INFO:teuthology.orchestra.run.smithi187.stdout:Permissive 2021-11-20T05:18:58.977 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi058.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi187.front.sepia.ceph.com': 'permissive'} 2021-11-20T05:18:58.977 DEBUG:teuthology.orchestra.run.smithi058:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-20T05:18:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T05:18:59.012 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-20T05:18:59.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T05:18:59.045 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-11-20T05:18:59.046 DEBUG:teuthology.orchestra.run.smithi058:> sudo /usr/sbin/setenforce permissive 2021-11-20T05:18:59.085 DEBUG:teuthology.orchestra.run.smithi187:> sudo /usr/sbin/setenforce permissive 2021-11-20T05:18:59.121 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-11-20T05:18:59.133 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-11-20T05:18:59.134 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-11-20T05:18:59.161 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-11-20T05:18:59.162 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi058.front.sepia.ceph.com,smithi187.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-11-20T05:18:59.532 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-11-20T05:19:00.325 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-11-20T05:19:03.264 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-11-20T05:19:03.388 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:03 +0000 (0:00:00.141) 0:00:00.141 ***** 2021-11-20T05:19:03.402 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:03 +0000 (0:00:00.012) 0:00:00.154 ***** 2021-11-20T05:19:04.852 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi187.front.sepia.ceph.com] 2021-11-20T05:19:04.899 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:04 +0000 (0:00:01.498) 0:00:01.652 ***** 2021-11-20T05:19:04.910 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] 2021-11-20T05:19:04.958 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:04 +0000 (0:00:00.058) 0:00:01.710 ***** 2021-11-20T05:19:05.163 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-11-20T05:19:05.165 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-20T05:19:05.190 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:05 +0000 (0:00:00.232) 0:00:01.943 ***** 2021-11-20T05:19:05.197 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-20T05:19:05.221 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:05 +0000 (0:00:00.030) 0:00:01.973 ***** 2021-11-20T05:19:05.717 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-11-20T05:19:05.719 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-11-20T05:19:05.741 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:05 +0000 (0:00:00.520) 0:00:02.494 ***** 2021-11-20T05:19:05.750 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-11-20T05:19:05.775 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:05 +0000 (0:00:00.033) 0:00:02.527 ***** 2021-11-20T05:19:05.813 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-11-20T05:19:05.816 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:19:05.837 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:05 +0000 (0:00:00.061) 0:00:02.589 ***** 2021-11-20T05:19:05.855 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:19:05.878 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:05 +0000 (0:00:00.041) 0:00:02.631 ***** 2021-11-20T05:19:06.118 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-11-20T05:19:06.120 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:19:06.142 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:06 +0000 (0:00:00.264) 0:00:02.895 ***** 2021-11-20T05:19:06.150 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:19:06.174 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:06 +0000 (0:00:00.030) 0:00:02.926 ***** 2021-11-20T05:19:07.060 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-11-20T05:19:07.062 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637385546.1868305-20976-169649158113812/source", "state": "file", "uid": 0} 2021-11-20T05:19:07.062 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637385546.2199929-20981-177179292722342/source", "state": "file", "uid": 0} 2021-11-20T05:19:07.084 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:07 +0000 (0:00:00.910) 0:00:03.837 ***** 2021-11-20T05:19:07.104 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:07 +0000 (0:00:00.019) 0:00:03.856 ***** 2021-11-20T05:19:07.645 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-11-20T05:19:07.647 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-11-20T05:19:07.648 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-11-20T05:19:07.667 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-11-20T05:19:07.669 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-11-20T05:19:07.744 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-11-20T05:19:07.746 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-11-20T05:19:07.760 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-11-20T05:19:07.762 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:07 +0000 (0:00:00.656) 0:00:04.513 ***** 2021-11-20T05:19:08.295 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] 2021-11-20T05:19:08.306 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] 2021-11-20T05:19:08.323 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** 2021-11-20T05:19:08.325 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:08 +0000 (0:00:00.563) 0:00:05.076 ***** 2021-11-20T05:19:08.349 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-11-20T05:19:08.360 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-11-20T05:19:08.367 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-11-20T05:19:08.522 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:08 +0000 (0:00:00.198) 0:00:05.275 ***** 2021-11-20T05:19:08.541 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:08 +0000 (0:00:00.018) 0:00:05.294 ***** 2021-11-20T05:19:09.191 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi058.front.sepia.ceph.com] 2021-11-20T05:19:09.200 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] 2021-11-20T05:19:09.261 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Saturday 20 November 2021 05:19:09 +0000 (0:00:00.720) 0:00:06.014 ***** 2021-11-20T05:19:09.284 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:09 +0000 (0:00:00.022) 0:00:06.037 ***** 2021-11-20T05:19:09.912 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.180800", "end": "2021-11-20 05:19:09.887859", "rc": 0, "start": "2021-11-20 05:19:09.707059", "stderr": "", "stderr_lines": [], "stdout": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "stdout_lines": ["d72e06d3c78c8d9639a8ca75effa043ee468910f"]} 2021-11-20T05:19:09.935 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:09 +0000 (0:00:00.650) 0:00:06.687 ***** 2021-11-20T05:19:09.943 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.183427", "end": "2021-11-20 05:19:09.915090", "rc": 0, "start": "2021-11-20 05:19:09.731663", "stderr": "", "stderr_lines": [], "stdout": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "stdout_lines": ["d72e06d3c78c8d9639a8ca75effa043ee468910f"]} 2021-11-20T05:19:09.968 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:09 +0000 (0:00:00.033) 0:00:06.721 ***** 2021-11-20T05:19:10.147 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.014080", "end": "2021-11-20 05:19:10.129899", "failed_when_result": false, "rc": 0, "start": "2021-11-20 05:19:10.115819", "stderr": "", "stderr_lines": [], "stdout": "2f6366fd31821994f3d217ad18c1675bf42a2115", "stdout_lines": ["2f6366fd31821994f3d217ad18c1675bf42a2115"]} 2021-11-20T05:19:10.176 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:10 +0000 (0:00:00.207) 0:00:06.928 ***** 2021-11-20T05:19:10.194 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.018131", "end": "2021-11-20 05:19:10.163343", "failed_when_result": false, "rc": 0, "start": "2021-11-20 05:19:10.145212", "stderr": "", "stderr_lines": [], "stdout": "2f6366fd31821994f3d217ad18c1675bf42a2115", "stdout_lines": ["2f6366fd31821994f3d217ad18c1675bf42a2115"]} 2021-11-20T05:19:10.219 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:10 +0000 (0:00:00.042) 0:00:06.970 ***** 2021-11-20T05:19:10.246 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:19:10.271 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:10 +0000 (0:00:00.053) 0:00:07.023 ***** 2021-11-20T05:19:10.290 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:19:10.314 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:10 +0000 (0:00:00.043) 0:00:07.067 ***** 2021-11-20T05:19:10.342 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-11-20T05:19:10.344 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:19:10.366 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:10 +0000 (0:00:00.051) 0:00:07.118 ***** 2021-11-20T05:19:10.383 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:19:10.407 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:10 +0000 (0:00:00.041) 0:00:07.160 ***** 2021-11-20T05:19:10.436 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-11-20T05:19:10.437 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:19:10.459 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:10 +0000 (0:00:00.051) 0:00:07.211 ***** 2021-11-20T05:19:10.477 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:19:10.503 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:10 +0000 (0:00:00.043) 0:00:07.255 ***** 2021-11-20T05:19:10.530 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-11-20T05:19:10.532 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:19:10.553 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:10 +0000 (0:00:00.050) 0:00:07.306 ***** 2021-11-20T05:19:10.570 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:19:10.594 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:10 +0000 (0:00:00.040) 0:00:07.347 ***** 2021-11-20T05:19:10.635 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:19:10.665 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:10 +0000 (0:00:00.070) 0:00:07.417 ***** 2021-11-20T05:19:10.673 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:19:10.697 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:10 +0000 (0:00:00.031) 0:00:07.449 ***** 2021-11-20T05:19:10.913 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-11-20T05:19:10.915 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:19:10.938 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:10 +0000 (0:00:00.241) 0:00:07.690 ***** 2021-11-20T05:19:10.946 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:19:10.971 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:19:10 +0000 (0:00:00.032) 0:00:07.723 ***** 2021-11-20T05:19:11.234 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-11-20T05:19:11.236 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-11-20T05:19:11.256 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-11-20T05:19:11.435 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-11-20T05:19:11.458 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-11-20T05:19:11.649 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-11-20T05:19:11.672 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-11-20T05:19:11.862 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-11-20T05:19:11.865 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-11-20T05:19:12.066 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-11-20T05:19:12.069 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-11-20T05:19:12.261 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-11-20T05:19:12.273 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-11-20T05:19:12.466 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-11-20T05:19:12.487 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-11-20T05:19:12.662 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-11-20T05:19:12.695 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-11-20T05:19:12.856 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-11-20T05:19:12.899 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-11-20T05:19:13.056 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-11-20T05:19:13.110 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-11-20T05:19:13.258 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-11-20T05:19:13.312 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-11-20T05:19:13.450 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-11-20T05:19:13.525 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-11-20T05:19:13.653 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-11-20T05:19:13.726 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-11-20T05:19:13.845 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-11-20T05:19:13.942 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-11-20T05:19:14.048 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-11-20T05:19:14.144 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-11-20T05:19:14.253 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-11-20T05:19:14.351 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-11-20T05:19:14.446 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-11-20T05:19:14.562 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-11-20T05:19:14.648 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-11-20T05:19:14.776 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-11-20T05:19:14.852 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-11-20T05:19:14.995 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-11-20T05:19:15.050 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-11-20T05:19:15.209 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-11-20T05:19:15.252 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-11-20T05:19:15.411 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-11-20T05:19:15.455 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-11-20T05:19:15.614 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-11-20T05:19:15.647 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-11-20T05:19:15.819 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-11-20T05:19:15.841 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-11-20T05:19:16.030 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-11-20T05:19:16.042 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-11-20T05:19:16.233 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-11-20T05:19:16.245 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-11-20T05:19:16.434 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-11-20T05:19:16.445 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-11-20T05:19:16.635 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-11-20T05:19:16.657 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-11-20T05:19:16.839 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-11-20T05:19:16.862 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-11-20T05:19:17.031 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-11-20T05:19:17.074 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-11-20T05:19:17.232 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-11-20T05:19:17.285 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-11-20T05:19:17.421 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-11-20T05:19:17.485 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-11-20T05:19:17.622 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-11-20T05:19:17.706 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-11-20T05:19:17.825 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-11-20T05:19:17.909 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-11-20T05:19:18.025 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-20T05:19:18.120 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-11-20T05:19:18.226 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-11-20T05:19:18.332 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-20T05:19:18.417 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-11-20T05:19:18.533 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-11-20T05:19:18.629 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-11-20T05:19:18.745 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-11-20T05:19:18.820 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-11-20T05:19:18.957 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-11-20T05:19:19.021 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-11-20T05:19:19.158 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-11-20T05:19:19.211 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-11-20T05:19:19.360 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-11-20T05:19:19.413 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-11-20T05:19:19.571 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-11-20T05:19:19.615 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-20T05:19:19.786 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-11-20T05:19:19.809 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-11-20T05:19:19.997 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-20T05:19:20.030 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-11-20T05:19:20.221 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-11-20T05:19:20.243 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-11-20T05:19:20.427 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-11-20T05:19:20.439 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-11-20T05:19:20.639 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-11-20T05:19:20.641 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-11-20T05:19:20.834 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-11-20T05:19:20.853 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-11-20T05:19:21.033 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-11-20T05:19:21.055 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-11-20T05:19:21.226 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-11-20T05:19:21.269 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-11-20T05:19:21.431 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-11-20T05:19:21.474 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-11-20T05:19:21.622 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-11-20T05:19:21.688 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-11-20T05:19:21.830 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-11-20T05:19:21.908 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-11-20T05:19:22.037 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-11-20T05:19:22.101 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-11-20T05:19:22.241 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-11-20T05:19:22.315 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-11-20T05:19:22.431 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-11-20T05:19:22.521 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-11-20T05:19:22.628 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-11-20T05:19:22.727 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-11-20T05:19:22.840 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-11-20T05:19:22.946 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-11-20T05:19:23.029 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-20T05:19:23.158 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-11-20T05:19:23.234 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-11-20T05:19:23.359 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-11-20T05:19:23.434 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-11-20T05:19:23.563 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-20T05:19:23.628 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-11-20T05:19:23.778 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-11-20T05:19:23.832 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-11-20T05:19:23.984 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-11-20T05:19:24.028 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-11-20T05:19:24.186 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-11-20T05:19:24.229 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-11-20T05:19:24.387 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-11-20T05:19:24.419 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-11-20T05:19:24.601 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-11-20T05:19:24.623 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-11-20T05:19:24.804 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-11-20T05:19:24.816 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-11-20T05:19:25.019 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-11-20T05:19:25.020 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-11-20T05:19:25.221 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-11-20T05:19:25.351 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-11-20T05:19:25.414 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-11-20T05:19:25.437 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-11-20T05:19:25.616 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-11-20T05:19:25.638 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-11-20T05:19:25.817 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-11-20T05:19:25.860 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-11-20T05:19:26.011 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-11-20T05:19:26.075 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-11-20T05:19:26.201 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-11-20T05:19:26.275 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-11-20T05:19:26.415 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-11-20T05:19:26.490 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-11-20T05:19:26.605 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-11-20T05:19:26.691 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-11-20T05:19:26.796 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-11-20T05:19:26.894 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-11-20T05:19:27.000 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-11-20T05:19:27.106 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-11-20T05:19:27.202 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-11-20T05:19:27.319 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-11-20T05:19:27.394 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-11-20T05:19:27.520 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-11-20T05:19:27.586 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-11-20T05:19:27.733 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-11-20T05:19:27.788 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-11-20T05:19:27.946 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-11-20T05:19:27.989 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-11-20T05:19:28.148 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-11-20T05:19:28.180 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-11-20T05:19:28.359 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-11-20T05:19:28.381 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-11-20T05:19:28.571 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-11-20T05:19:28.583 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-11-20T05:19:28.783 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-11-20T05:19:28.784 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-11-20T05:19:28.982 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-11-20T05:19:28.994 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-11-20T05:19:29.172 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-11-20T05:19:29.194 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-11-20T05:19:29.372 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-11-20T05:19:29.405 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-11-20T05:19:29.575 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-11-20T05:19:29.608 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-11-20T05:19:29.777 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-11-20T05:19:29.821 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-11-20T05:19:29.981 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-11-20T05:19:30.045 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-11-20T05:19:30.173 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-20T05:19:30.247 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-11-20T05:19:30.375 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-11-20T05:19:30.449 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-11-20T05:19:30.566 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-11-20T05:19:30.662 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-11-20T05:19:30.768 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-11-20T05:19:30.863 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-11-20T05:19:30.969 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-11-20T05:19:31.077 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-20T05:19:31.172 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-11-20T05:19:31.279 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-11-20T05:19:31.364 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-11-20T05:19:31.492 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-11-20T05:19:31.556 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-11-20T05:19:31.704 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-11-20T05:19:31.768 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-11-20T05:19:31.914 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-11-20T05:19:31.962 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-11-20T05:19:32.122 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-11-20T05:19:32.155 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-11-20T05:19:32.336 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-11-20T05:19:32.369 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-11-20T05:19:32.547 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-11-20T05:19:32.568 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-11-20T05:19:32.748 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-11-20T05:19:32.770 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-11-20T05:19:32.961 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-11-20T05:19:32.972 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-11-20T05:19:33.163 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-11-20T05:19:33.175 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-11-20T05:19:33.378 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-11-20T05:19:33.389 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-11-20T05:19:33.577 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-11-20T05:19:33.580 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-11-20T05:19:33.779 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-11-20T05:19:33.793 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-11-20T05:19:33.982 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-11-20T05:19:33.993 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-11-20T05:19:34.173 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-11-20T05:19:34.206 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-11-20T05:19:34.374 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-11-20T05:19:34.407 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-11-20T05:19:34.578 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-11-20T05:19:34.618 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-11-20T05:19:34.765 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-11-20T05:19:34.828 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-11-20T05:19:34.966 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-11-20T05:19:35.030 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-11-20T05:19:35.157 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-11-20T05:19:35.231 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-11-20T05:19:35.358 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-11-20T05:19:35.443 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-11-20T05:19:35.559 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-11-20T05:19:35.656 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-11-20T05:19:35.752 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-11-20T05:19:35.858 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-11-20T05:19:35.948 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-11-20T05:19:36.078 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-11-20T05:19:36.162 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-11-20T05:19:36.277 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-11-20T05:19:36.351 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-11-20T05:19:36.491 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-11-20T05:19:36.555 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-11-20T05:19:36.694 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-11-20T05:19:36.747 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-11-20T05:19:36.904 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-11-20T05:19:36.947 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-11-20T05:19:37.106 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-11-20T05:19:37.150 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-11-20T05:19:37.318 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-11-20T05:19:37.340 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-11-20T05:19:37.537 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-11-20T05:19:37.547 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-11-20T05:19:37.736 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-11-20T05:19:37.946 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-11-20T05:19:38.156 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-11-20T05:19:38.365 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-11-20T05:19:38.564 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-11-20T05:19:38.775 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-11-20T05:19:39.165 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-11-20T05:19:40.473 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-11-20T05:19:40.905 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-11-20T05:19:42.431 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-11-20T05:19:42.570 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-11-20T05:19:44.032 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-11-20T05:19:44.169 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-11-20T05:19:45.611 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-11-20T05:19:45.677 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-11-20T05:19:47.148 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-11-20T05:19:47.295 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-11-20T05:19:48.793 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-11-20T05:19:48.836 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-11-20T05:19:50.368 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-11-20T05:19:50.494 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-11-20T05:19:51.984 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-11-20T05:19:52.103 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-11-20T05:19:53.534 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-11-20T05:19:53.682 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-11-20T05:19:55.196 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-11-20T05:19:55.280 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-11-20T05:19:56.921 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-11-20T05:19:56.977 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-11-20T05:19:58.612 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-11-20T05:19:58.634 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-11-20T05:20:00.274 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-11-20T05:20:00.358 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-11-20T05:20:01.852 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-11-20T05:20:02.094 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-11-20T05:20:03.468 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "name": "melissali", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-11-20T05:20:03.739 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-11-20T05:20:05.100 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-11-20T05:20:05.485 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "name": "melissali", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-11-20T05:20:06.789 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-11-20T05:20:07.238 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-11-20T05:20:08.415 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-11-20T05:20:08.976 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-11-20T05:20:10.159 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10121, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10121} 2021-11-20T05:20:10.673 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-11-20T05:20:11.896 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10122, "groups": "sudo", "home": "/home/choffman", "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "name": "choffman", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10122} 2021-11-20T05:20:12.059 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:20:12 +0000 (0:01:01.088) 0:01:08.811 ***** 2021-11-20T05:20:12.128 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-11-20T05:20:12.152 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:20:12 +0000 (0:00:00.093) 0:01:08.905 ***** 2021-11-20T05:20:12.248 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-11-20T05:20:12.251 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@desktop 2021-11-20T05:20:12.252 INFO:teuthology.task.ansible.out: QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2021-11-20T05:20:12.254 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed20 2021-11-20T05:20:12.256 INFO:teuthology.task.ansible.out:82caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name" 2021-11-20T05:20:12.258 INFO:teuthology.task.ansible.out:: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915 2021-11-20T05:20:12.259 INFO:teuthology.task.ansible.out:d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZ 2021-11-20T05:20:12.261 INFO:teuthology.task.ansible.out:YM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a9 2021-11-20T05:20:12.261 INFO:teuthology.task.ansible.out:5579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 2021-11-20T05:20:12.263 INFO:teuthology.task.ansible.out: 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp 2021-11-20T05:20:12.265 INFO:teuthology.task.ansible.out:", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3 2021-11-20T05:20:12.266 INFO:teuthology.task.ansible.out:624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9Jj 2021-11-20T05:20:12.268 INFO:teuthology.task.ansible.out:enDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, 2021-11-20T05:20:12.269 INFO:teuthology.task.ansible.out: {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "sh 2021-11-20T05:20:12.270 INFO:teuthology.task.ansible.out:yamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a 2021-11-20T05:20:12.272 INFO:teuthology.task.ansible.out:28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": " 2021-11-20T05:20:12.273 INFO:teuthology.task.ansible.out:zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}]}, "changed": false} 2021-11-20T05:20:12.296 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:20:12 +0000 (0:00:00.143) 0:01:09.049 ***** 2021-11-20T05:20:12.449 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-11-20T05:20:12.451 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10121, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10121} 2021-11-20T05:20:13.218 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-11-20T05:20:13.219 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"after": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "attempts": 1, "before": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "changed": false, "remote_url_changed": false} 2021-11-20T05:20:13.240 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:20:13 +0000 (0:00:00.944) 0:01:09.993 ***** 2021-11-20T05:20:13.573 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-11-20T05:20:13.774 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-11-20T05:20:13.986 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-11-20T05:20:14.133 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-11-20T05:20:14.135 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10122, "groups": "sudo", "home": "/home/choffman", "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "name": "choffman", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10122} 2021-11-20T05:20:14.260 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T05:20:14.262 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-11-20T05:20:14.264 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-11-20T05:20:14.283 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:20:14 +0000 (0:00:01.042) 0:01:11.035 ***** 2021-11-20T05:20:14.412 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-11-20T05:20:14.496 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-11-20T05:20:14.519 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:20:14 +0000 (0:00:00.236) 0:01:11.272 ***** 2021-11-20T05:20:14.612 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-11-20T05:20:14.615 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@desktop 2021-11-20T05:20:14.617 INFO:teuthology.task.ansible.out: QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2021-11-20T05:20:14.618 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed20 2021-11-20T05:20:14.620 INFO:teuthology.task.ansible.out:82caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name" 2021-11-20T05:20:14.621 INFO:teuthology.task.ansible.out:: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9eb 2021-11-20T05:20:14.622 INFO:teuthology.task.ansible.out:ba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915 2021-11-20T05:20:14.623 INFO:teuthology.task.ansible.out:d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZ 2021-11-20T05:20:14.625 INFO:teuthology.task.ansible.out:YM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 2021-11-20T05:20:14.627 INFO:teuthology.task.ansible.out: 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp 2021-11-20T05:20:14.628 INFO:teuthology.task.ansible.out:", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3 2021-11-20T05:20:14.630 INFO:teuthology.task.ansible.out:624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9Jj 2021-11-20T05:20:14.632 INFO:teuthology.task.ansible.out:enDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432 2021-11-20T05:20:14.632 INFO:teuthology.task.ansible.out:aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "sh 2021-11-20T05:20:14.634 INFO:teuthology.task.ansible.out:yamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a 2021-11-20T05:20:14.636 INFO:teuthology.task.ansible.out:28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": " 2021-11-20T05:20:14.637 INFO:teuthology.task.ansible.out:zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}]}, "changed": false} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "ke 2021-11-20T05:20:14.639 INFO:teuthology.task.ansible.out:y": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-11-20T05:20:14.662 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:20:14 +0000 (0:00:00.142) 0:01:11.414 ***** 2021-11-20T05:20:14.827 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-11-20T05:20:15.042 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-11-20T05:20:15.254 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-11-20T05:20:15.265 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:15.339 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-11-20T05:20:15.341 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"after": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "attempts": 1, "before": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "changed": false, "remote_url_changed": false} 2021-11-20T05:20:15.363 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:20:15 +0000 (0:00:00.701) 0:01:12.116 ***** 2021-11-20T05:20:15.474 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T05:20:15.476 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-11-20T05:20:15.478 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-11-20T05:20:15.708 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-11-20T05:20:15.709 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-11-20T05:20:15.719 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-11-20T05:20:15.720 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:15.731 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:15.932 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-11-20T05:20:15.954 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-11-20T05:20:16.156 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-11-20T05:20:16.158 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJ 2021-11-20T05:20:16.160 INFO:teuthology.task.ansible.out:E8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-11-20T05:20:16.376 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-11-20T05:20:16.418 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-11-20T05:20:16.420 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-11-20T05:20:16.587 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-11-20T05:20:16.629 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-11-20T05:20:16.788 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-11-20T05:20:16.989 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-11-20T05:20:16.990 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-11-20T05:20:17.001 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-11-20T05:20:17.002 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-11-20T05:20:17.221 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-11-20T05:20:17.223 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-11-20T05:20:17.224 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-11-20T05:20:17.434 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-11-20T05:20:17.445 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-11-20T05:20:17.633 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-11-20T05:20:17.666 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-11-20T05:20:17.677 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:17.846 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-11-20T05:20:17.849 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-11-20T05:20:17.888 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-11-20T05:20:17.890 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-11-20T05:20:18.056 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-11-20T05:20:18.120 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-11-20T05:20:18.121 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-11-20T05:20:18.131 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:18.133 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:18.268 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-11-20T05:20:18.353 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-11-20T05:20:18.470 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-11-20T05:20:18.580 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-11-20T05:20:18.583 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-11-20T05:20:18.676 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-11-20T05:20:18.804 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-11-20T05:20:18.888 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-11-20T05:20:19.015 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-11-20T05:20:19.100 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-11-20T05:20:19.102 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-11-20T05:20:19.248 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-11-20T05:20:19.321 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-11-20T05:20:19.459 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-11-20T05:20:19.461 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-11-20T05:20:19.521 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-11-20T05:20:19.677 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-11-20T05:20:19.679 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-11-20T05:20:19.731 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-11-20T05:20:19.888 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-11-20T05:20:19.940 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-11-20T05:20:20.107 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-11-20T05:20:20.160 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-20T05:20:20.330 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-11-20T05:20:20.332 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-11-20T05:20:20.372 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-11-20T05:20:20.550 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-11-20T05:20:20.572 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-11-20T05:20:20.772 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-11-20T05:20:20.784 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-11-20T05:20:20.785 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-11-20T05:20:20.983 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-11-20T05:20:20.994 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-11-20T05:20:20.996 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-11-20T05:20:21.204 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-11-20T05:20:21.216 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-11-20T05:20:21.414 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-11-20T05:20:21.426 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-11-20T05:20:21.624 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-11-20T05:20:21.646 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-11-20T05:20:21.648 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-11-20T05:20:21.826 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-11-20T05:20:21.828 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-20T05:20:21.858 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-11-20T05:20:22.045 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-11-20T05:20:22.047 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-11-20T05:20:22.049 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-11-20T05:20:22.088 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-11-20T05:20:22.245 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-11-20T05:20:22.298 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-11-20T05:20:22.468 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-11-20T05:20:22.531 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-11-20T05:20:22.668 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-11-20T05:20:22.753 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-20T05:20:22.880 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-11-20T05:20:22.964 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-11-20T05:20:23.090 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-11-20T05:20:23.184 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-11-20T05:20:23.289 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-11-20T05:20:23.290 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-11-20T05:20:23.404 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-11-20T05:20:23.406 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-11-20T05:20:23.500 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-11-20T05:20:23.624 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-11-20T05:20:23.626 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-11-20T05:20:23.709 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-11-20T05:20:23.834 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-11-20T05:20:23.918 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-11-20T05:20:23.920 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-11-20T05:20:24.054 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-11-20T05:20:24.118 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-11-20T05:20:24.264 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-11-20T05:20:24.327 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-11-20T05:20:24.494 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-11-20T05:20:24.496 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-20T05:20:24.536 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-11-20T05:20:24.538 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-11-20T05:20:24.704 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-11-20T05:20:24.707 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-11-20T05:20:24.708 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-11-20T05:20:24.747 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-11-20T05:20:24.926 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-11-20T05:20:24.958 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-11-20T05:20:25.147 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-11-20T05:20:25.169 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-11-20T05:20:25.367 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-11-20T05:20:25.378 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-20T05:20:25.576 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-11-20T05:20:25.587 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-11-20T05:20:25.798 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-11-20T05:20:25.800 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-11-20T05:20:26.002 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-11-20T05:20:26.014 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-11-20T05:20:26.016 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-11-20T05:20:26.201 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-11-20T05:20:26.234 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-11-20T05:20:26.417 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-11-20T05:20:26.449 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-11-20T05:20:26.636 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-11-20T05:20:26.669 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-11-20T05:20:26.672 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-11-20T05:20:26.840 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-11-20T05:20:26.883 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-11-20T05:20:27.049 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-11-20T05:20:27.103 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-11-20T05:20:27.249 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-11-20T05:20:27.323 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-11-20T05:20:27.324 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-11-20T05:20:27.459 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-11-20T05:20:27.543 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-11-20T05:20:27.672 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-11-20T05:20:27.755 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-11-20T05:20:27.881 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-11-20T05:20:27.966 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-11-20T05:20:28.081 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-11-20T05:20:28.188 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-20T05:20:28.294 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-11-20T05:20:28.400 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-11-20T05:20:28.505 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-11-20T05:20:28.622 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-11-20T05:20:28.706 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-11-20T05:20:28.831 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-11-20T05:20:28.916 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-11-20T05:20:28.918 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-11-20T05:20:29.052 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-11-20T05:20:29.136 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-11-20T05:20:29.262 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-11-20T05:20:29.336 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-11-20T05:20:29.482 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-11-20T05:20:29.546 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-11-20T05:20:29.691 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-11-20T05:20:29.745 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-11-20T05:20:29.747 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-11-20T05:20:29.913 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-11-20T05:20:29.956 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-11-20T05:20:30.127 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-11-20T05:20:30.171 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-11-20T05:20:30.348 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-11-20T05:20:30.380 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-11-20T05:20:30.568 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-11-20T05:20:30.601 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-11-20T05:20:30.789 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-11-20T05:20:30.801 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-11-20T05:20:31.009 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-11-20T05:20:31.012 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-11-20T05:20:31.231 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-11-20T05:20:31.242 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-11-20T05:20:31.452 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-11-20T05:20:31.454 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-11-20T05:20:31.458 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-11-20T05:20:31.654 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-11-20T05:20:31.666 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-11-20T05:20:31.866 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-11-20T05:20:31.877 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-11-20T05:20:31.879 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-11-20T05:20:32.076 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-11-20T05:20:32.098 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-11-20T05:20:32.287 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-11-20T05:20:32.309 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-11-20T05:20:32.488 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-11-20T05:20:32.531 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-11-20T05:20:32.702 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-11-20T05:20:32.765 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-11-20T05:20:32.767 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-11-20T05:20:32.901 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-11-20T05:20:32.903 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-20T05:20:33.006 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-11-20T05:20:33.123 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-11-20T05:20:33.208 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-11-20T05:20:33.323 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-11-20T05:20:33.428 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-11-20T05:20:33.533 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-11-20T05:20:33.650 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-11-20T05:20:33.735 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-11-20T05:20:33.861 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-11-20T05:20:33.945 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-11-20T05:20:34.081 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-11-20T05:20:34.155 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-11-20T05:20:34.291 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-11-20T05:20:34.364 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-11-20T05:20:34.512 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-11-20T05:20:34.514 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-11-20T05:20:34.564 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-11-20T05:20:34.730 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-11-20T05:20:34.782 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-11-20T05:20:34.971 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-11-20T05:20:34.992 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-11-20T05:20:35.182 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-11-20T05:20:35.203 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-11-20T05:20:35.402 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-11-20T05:20:35.404 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-11-20T05:20:35.614 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "u 2021-11-20T05:20:35.616 INFO:teuthology.task.ansible.out:ser": "rakesh", "validate_certs": true} 2021-11-20T05:20:35.812 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-11-20T05:20:35.836 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-11-20T05:20:36.034 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-11-20T05:20:36.047 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-11-20T05:20:36.048 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-20T05:20:36.238 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-11-20T05:20:36.259 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-11-20T05:20:36.447 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-11-20T05:20:36.490 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-11-20T05:20:36.660 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-11-20T05:20:36.703 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-11-20T05:20:36.873 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-11-20T05:20:36.875 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-11-20T05:20:36.926 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-11-20T05:20:37.081 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-11-20T05:20:37.135 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-11-20T05:20:37.291 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-11-20T05:20:37.356 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-11-20T05:20:37.493 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-11-20T05:20:37.591 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-11-20T05:20:37.707 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-11-20T05:20:37.801 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-11-20T05:20:37.905 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-11-20T05:20:38.021 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-11-20T05:20:38.115 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-11-20T05:20:38.240 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-11-20T05:20:38.325 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-11-20T05:20:38.450 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-11-20T05:20:38.524 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-11-20T05:20:38.672 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-11-20T05:20:38.735 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-11-20T05:20:38.881 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-11-20T05:20:38.955 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-11-20T05:20:39.101 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-11-20T05:20:39.165 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-11-20T05:20:39.321 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-11-20T05:20:39.375 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-11-20T05:20:39.542 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-11-20T05:20:39.575 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4m0LAcC7XiFiXFtYcCvZqX1CtyV9JU+VUEawZJpWrGTOr8c+vIexq7+WKg+eIUMot4+dtfwG8lByS1VofglIe5/quv17Xj7dO/hF2WkV8O7RYt7NhzmrKg54yB73pbRpvqzGceWWDvGRsWQGzbY7ARqOp4FPIDJvgXABCkak+hmjXyXpvXN418CBDbDmxHWwFb9QtB/MpQFiFrfl1EMDj8DaXZNWgo5J/qytic6LgiyPHhHfhSylk4cl9TGftNatobpMjqBT0O2/plk/qf+7pHPuvGSgihjijdpEPzVfctXE2OmsQAIZgkzXeCqts+s6ZYdUKq1FvpV5rMM8ZLqfCkDjEzFENJS68YYE2EjxAEgMzj7h/sWc4B5I5pxkAqVAQSGjYRu6rCCbFz8PXUdTyW+Vs1YeylnlBMiCa4xE4gwvW3HlWEnumtyURd1PRENrGJsJ/0uX6bpltbSg4sGACRYpLPJMm3OhqI3Jm/J/e2NLjfCw+svU2447xnfJOh68= pdhange@vaquita\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCoeyVaMjhcGrWQB+4Rq+SSIi8+OY/kx2s1vofiGt1gu3RNtFmG0Df1VITJJ32dbzaW7kTNxaRvhldNd/n7yEMssG+Wg+a5WKvSudlbVMmscbSXdFuVbcCW1jNrTGCYrxL7Sn5e7NQEQfAsCbfXwTcYBw33NdBVovC+gs2V3sZGvX3Y0akZONNuVg1mhu0L4m1/5IegJh005KeilwetHA2A5Dr10RG6oyIgT7t4z5gZXsrOiG3UQBXaekILYV9jxMWoFXGkVLc8TpL9SUAdxE3Owc0u1XiPqYYlEN+zDvE0554Zj8Oc2ZO6HRXCZXQFvzXeCD12YswcOfkIDmJRF7GT prashant@mauli", "key_options": null, "k 2021-11-20T05:20:39.576 INFO:teuthology.task.ansible.out:eyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-11-20T05:20:39.752 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-11-20T05:20:39.784 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-11-20T05:20:39.972 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-11-20T05:20:39.994 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-11-20T05:20:40.193 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-11-20T05:20:40.196 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-11-20T05:20:40.204 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-11-20T05:20:40.206 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-11-20T05:20:40.402 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-11-20T05:20:40.413 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCt4sQTrDDg562IWY9Bw2fxFR7Y7b5lfo9egHa1qNFR5K4gt8lUwQXgzcxZK5dgSTZXHDGxrbYKWsdb8NbpgU/YjSqDUIptMZrhTI6pdHKZ2usY+nieaJj0NIZP/S7M93BbJeoNBYACjvDUr+vXEg6ZL3uAj+zFXr/4te+p4NqJBl4bSpvjcRK12O+F6HE3Q93j0Vlfs8IbXP4vcPcqOW+2usXw7JPdfjwuACJxrLTx/0mBsv8qXJDfOpPIHAMA/9SdTAzhs6XMo+fOwVOKkV+ki2fO3sl8pE16aFOkI3cYX36nkYYpZ2bJLnIPcLv37i3Nypl8eNLA4gzjkDf1o/KLO5vXOs76H189WGLqjh0SHAx/3KO6mrIhTq8Bv6YWoT3ntg22fXaxeajNhHcQ0bKZtGOAb0W1FcMRbSTkgEFAassYxlpTtuaCDIuA4fUVKW97Zu62eDDsSUtTWak/Yf940qAbptPPAw4ljmrvZRcjM9y6mGU8lKi5H4QoEOfbPok= nia@fedora", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-11-20T05:20:40.622 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-11-20T05:20:40.624 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-11-20T05:20:40.831 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-11-20T05:20:40.842 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-11-20T05:20:41.045 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-11-20T05:20:41.056 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-11-20T05:20:41.243 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-11-20T05:20:41.275 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-11-20T05:20:41.452 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-11-20T05:20:41.494 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-11-20T05:20:41.661 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-11-20T05:20:41.714 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-11-20T05:20:41.869 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-11-20T05:20:41.934 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-11-20T05:20:42.077 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-11-20T05:20:42.297 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn 2021-11-20T05:20:42.299 INFO:teuthology.task.ansible.out:+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-11-20T05:20:42.359 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-11-20T05:20:42.495 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-11-20T05:20:42.579 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-11-20T05:20:42.694 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-11-20T05:20:42.798 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-11-20T05:20:42.904 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-11-20T05:20:43.007 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4m0LAcC7XiFiXFtYcCvZqX1CtyV9JU+VUEawZJpWrGTOr8c+vIexq7+WKg+eIUMot4+dtfwG8lByS1VofglIe5/quv17Xj7dO/hF2WkV8O7RYt7NhzmrKg54yB73pbRpvqzGceWWDvGRsWQGzbY7ARqOp4FPIDJvgXABCkak+hmjXyXpvXN418CBDbDmxHWwFb9QtB/MpQFiFrfl1EMDj8DaXZNWgo5J/qytic6LgiyPHhHfhSylk4cl9TGftNatobpMjqBT0O2/plk/qf+7pHPuvGSgihjijdpEPzVfctXE2OmsQAIZgkzXeCqts+s6ZYdUKq1FvpV5rMM8ZLqfCkDjEzFENJS68YYE2EjxAEgMzj7h/sWc4B5I5pxkAqVAQSGjYRu6rCCbFz8PXUdTyW+Vs1YeylnlBMiCa4xE4gwvW3HlWEnumtyURd1PRENrGJsJ/0uX6bpltbSg4sGACRYpLPJMm3OhqI3Jm/J/e2NLjfCw+svU2447xnfJOh68= pdhange@vaquita\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCoeyVaMjhcGrWQB+4Rq+SSIi8+OY/kx2s1vofiGt1gu3RNtFmG0Df1VITJJ32dbzaW7kTNxaRvhldNd/n7yEMssG+Wg+a5WKvSudlbVMmscbSXdFuVbcCW1jNrTGCYrxL7Sn5e7NQEQfAsCbfXwTcYBw33NdBVovC+gs2V3sZGvX3Y0akZONNuVg1mhu0L4m1/5IegJh005KeilwetHA2A5Dr10RG6oyIgT7t4z5gZXsrOiG3UQBXaekILYV9jxMWoFXGkVLc8TpL9SUAdxE3Owc0u1XiPqYYlEN+zDvE0554Zj8Oc2ZO6HRXCZXQFvzXeCD12YswcOfkIDmJRF7GT prashant@mauli", "key_options": null, "k 2021-11-20T05:20:43.009 INFO:teuthology.task.ansible.out:eyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-11-20T05:20:43.112 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-11-20T05:20:43.227 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-11-20T05:20:43.321 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-11-20T05:20:43.446 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-11-20T05:20:43.530 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-11-20T05:20:43.666 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-11-20T05:20:43.668 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-11-20T05:20:43.739 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-11-20T05:20:43.886 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCt4sQTrDDg562IWY9Bw2fxFR7Y7b5lfo9egHa1qNFR5K4gt8lUwQXgzcxZK5dgSTZXHDGxrbYKWsdb8NbpgU/YjSqDUIptMZrhTI6pdHKZ2usY+nieaJj0NIZP/S7M93BbJeoNBYACjvDUr+vXEg6ZL3uAj+zFXr/4te+p4NqJBl4bSpvjcRK12O+F6HE3Q93j0Vlfs8IbXP4vcPcqOW+2usXw7JPdfjwuACJxrLTx/0mBsv8qXJDfOpPIHAMA/9SdTAzhs6XMo+fOwVOKkV+ki2fO3sl8pE16aFOkI3cYX36nkYYpZ2bJLnIPcLv37i3Nypl8eNLA4gzjkDf1o/KLO5vXOs76H189WGLqjh0SHAx/3KO6mrIhTq8Bv6YWoT3ntg22fXaxeajNhHcQ0bKZtGOAb0W1FcMRbSTkgEFAassYxlpTtuaCDIuA4fUVKW97Zu62eDDsSUtTWak/Yf940qAbptPPAw4ljmrvZRcjM9y6mGU8lKi5H4QoEOfbPok= nia@fedora", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-11-20T05:20:43.949 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-11-20T05:20:44.095 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-11-20T05:20:44.159 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-11-20T05:20:44.314 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-11-20T05:20:44.357 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-11-20T05:20:44.534 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-11-20T05:20:44.566 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-11-20T05:20:44.755 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-11-20T05:20:44.787 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-11-20T05:20:44.974 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-11-20T05:20:45.006 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM7LPHiaPQXdE+yt9cn7UMuRr8k/LjSUmMrnrAqp8ySQtX6OA3/NcOsxg6FJ45chZqeoXwhsYEW+7pTInDqFlOlmVynR26lWnZfk9DjBJZcloUCn8y5MR9QitNYsOMs5KwCEFmBH6sP+2vCFocXBnKQcVZLRfip3MEG4LoWYfI4ir7fLmK1Hr5G+GhtMstXvVSG4VE4IgbU9/LRIOv7P9qFn4bRJoUSiS2dDbUxVIuneA9bIEnB9ZEVGahEzlUhx6qDSs0FvrnneCQ1ayOFpxClzjT5GuKk/J3LJvWAw8CAwDXCf3FzsWQsvb5ru5VSazUI9uA3o9wWkbLW2oZPgtzI8lmwbL+MT58T92tqaJOPID3IZDGjzNrRqU5BQYpvFYHpHRRiqV625og+ktYfXgO+dTbgAxScomqutydI4cyJusDtnWtljWnEocS+2hUSZpmB6B3Wj70TuuzsbYrcrJ/NgRpQNX4dwHCJl6vPT+xkoLh1imYAC7klf7cnrOyGcs= choffman@choffman.net", "key_options": null, "keyfile": "/home/choffman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "choffman", "validate_certs": true} 2021-11-20T05:20:45.199 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-11-20T05:20:45.220 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:20:45 +0000 (0:00:29.856) 0:01:41.973 ***** 2021-11-20T05:20:45.313 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-20T05:20:45.315 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:45.326 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:45.327 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:45.338 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:45.349 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:45.350 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:45.360 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:45.373 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:45.404 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T05:20:45.406 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-11-20T05:20:45.582 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-20T05:20:45.583 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-11-20T05:20:45.593 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:45.604 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:45.626 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T05:20:45.627 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-11-20T05:20:45.813 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-20T05:20:45.815 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-11-20T05:20:45.836 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T05:20:45.837 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-11-20T05:20:46.023 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-20T05:20:46.024 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfad 2021-11-20T05:20:46.025 INFO:teuthology.task.ansible.out:a28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.034 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.045 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.047 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.057 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.058 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4ba 2021-11-20T05:20:46.060 INFO:teuthology.task.ansible.out:af222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.069 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.080 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.082 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.091 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.102 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.104 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.114 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.114 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.126 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.136 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.138 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.148 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.149 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.160 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.161 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.171 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.182 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.183 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.193 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.195 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.204 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.215 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.217 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.226 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.237 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.239 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.249 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.250 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.260 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.271 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.273 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.283 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.285 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.295 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.296 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.306 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.317 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.318 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.328 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.339 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.341 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.350 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.352 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.362 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.373 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.374 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.384 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.386 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.395 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.406 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.408 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.418 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.419 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.429 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.440 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.442 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.452 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.463 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.474 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.476 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.485 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.487 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.497 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T05:20:46.498 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46 2021-11-20T05:20:46.500 INFO:teuthology.task.ansible.out:e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.511 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.522 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.524 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.533 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.544 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.555 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.557 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.566 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.577 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.579 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.589 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.599 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.601 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.611 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.612 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.622 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.633 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.634 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.644 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.655 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.657 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.666 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.668 INFO:teuthology.task.ansible.out: skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.678 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.680 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.690 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.701 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.702 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.712 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.713 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.724 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.735 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.737 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.747 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.758 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.769 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.770 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.780 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.782 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.791 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.802 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.804 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.814 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.815 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.825 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.836 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.838 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.848 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.849 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.859 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.870 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.872 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.881 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.883 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.893 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.894 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.904 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.915 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.917 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.926 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.928 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.938 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T05:20:46.939 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalu 2021-11-20T05:20:46.941 INFO:teuthology.task.ansible.out:toiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.950 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.961 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.962 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:46.972 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:47.014 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:20:47 +0000 (0:00:01.793) 0:01:43.766 ***** 2021-11-20T05:20:47.082 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:47.102 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:20:47 +0000 (0:00:00.088) 0:01:43.855 ***** 2021-11-20T05:20:47.149 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T05:20:47.151 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-11-20T05:20:47.380 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-11-20T05:20:47.381 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi187.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-11-20T05:20:47.568 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-11-20T05:20:47.592 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T05:20:47.594 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-11-20T05:20:47.760 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:20:47.762 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-11-20T05:20:47.814 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T05:20:47.815 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-11-20T05:20:47.960 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:20:47.962 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-11-20T05:20:48.034 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-11-20T05:20:48.149 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi187.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-11-20T05:20:48.254 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T05:20:48.256 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-11-20T05:20:48.338 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:20:48.340 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-11-20T05:20:48.465 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T05:20:48.466 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-11-20T05:20:48.538 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:20:48.540 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-11-20T05:20:48.685 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T05:20:48.687 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM7LPHiaPQXdE+yt9cn7UMuRr8k/LjSUmMrnrAqp8ySQtX6OA3/NcOsxg6FJ45chZqeoXwhsYEW+7pTInDqFlOlmVynR26lWnZfk9DjBJZcloUCn8y5MR9QitNYsOMs5KwCEFmBH6sP+2vCFocXBnKQcVZLRfip3MEG4LoWYfI4ir7fLmK1Hr5G+GhtMstXvVSG4VE4IgbU9/LRIOv7P9qFn4bRJoUSiS2dDbUxVIuneA9bIEnB9ZEVGahEzlUhx6qDSs0FvrnneCQ1ayOFpxClzjT5GuKk/J3LJvWAw8CAwDXCf3FzsWQsvb5ru5VSazUI9uA3o9wWkbLW2oZPgtzI8lmwbL+MT58T92tqaJOPID3IZDGjzNrRqU5BQYpvFYHpHRRiqV625og+ktYfXgO+dTbgAxScomqutydI4cyJusDtnWtljWnEocS+2hUSZpmB6B3Wj70TuuzsbYrcrJ/NgRpQNX4dwHCJl6vPT+xkoLh1imYAC7klf7cnrOyGcs= choffman@choffman.net", "key_options": null, "keyfile": "/home/choffman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "choffman", "validate_certs": true} 2021-11-20T05:20:48.761 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:20:48.762 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-11-20T05:20:48.783 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:20:48 +0000 (0:00:01.680) 0:01:45.535 ***** 2021-11-20T05:20:48.865 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-20T05:20:48.867 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:48.877 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:48.888 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:48.889 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:48.899 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:48.901 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:48.910 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:48.921 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:20:48.923 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:48.933 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.110 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:20:49.112 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-11-20T05:20:49.154 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-20T05:20:49.156 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-11-20T05:20:49.166 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.168 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.303 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:20:49.304 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-11-20T05:20:49.398 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-20T05:20:49.400 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-11-20T05:20:49.504 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:20:49.506 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-11-20T05:20:49.611 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-20T05:20:49.612 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-11-20T05:20:49.622 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.633 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.634 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.644 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.646 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.656 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.667 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.669 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.678 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.689 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.691 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} TASK [Remove revoked users] **************************************************** ok: [smithi187.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-11-20T05:20:49.702 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-20T05:20:49.704 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.713 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.724 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.726 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.736 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.737 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.747 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.748 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.759 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.770 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.781 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.783 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.793 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.804 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.805 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.815 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.826 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.828 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.838 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.849 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.860 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.862 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.872 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.873 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.883 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.884 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi187.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-11-20T05:20:49.895 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.906 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.917 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.919 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.928 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.930 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.940 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.951 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.952 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.962 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.964 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.973 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.984 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.986 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.996 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:49.997 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.007 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.018 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.019 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.029 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.040 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.042 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.052 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.053 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.063 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.074 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.077 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi187.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-11-20T05:20:50.086 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-20T05:20:50.088 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.098 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.109 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.120 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.122 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.132 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.134 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.144 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.155 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.156 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.166 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.169 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.178 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.188 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.190 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.200 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.211 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.222 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.223 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.233 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.234 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.244 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.255 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.257 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.266 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.268 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi187.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.279 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.293 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.302 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.313 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.315 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.324 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.335 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.337 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.346 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.348 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.358 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.369 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.370 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.380 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.381 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.391 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.402 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.404 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.413 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.424 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.426 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.436 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.447 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.458 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.459 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.460 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi187.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-11-20T05:20:50.471 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.473 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.483 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.484 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.494 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.505 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.507 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.517 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.519 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.528 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.529 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.543 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.554 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.555 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.566 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.577 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.617 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:20:50 +0000 (0:00:01.834) 0:01:47.369 ***** 2021-11-20T05:20:50.656 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:20:50.658 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-11-20T05:20:50.688 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-11-20T05:20:50.690 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:20:50.711 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:20:50 +0000 (0:00:00.093) 0:01:47.463 ***** 2021-11-20T05:20:50.844 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:20:50.846 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-11-20T05:20:51.001 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-11-20T05:20:51.043 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-11-20T05:20:51.200 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-11-20T05:20:51.242 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-11-20T05:20:51.398 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-11-20T05:20:51.430 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-11-20T05:20:51.599 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-11-20T05:20:51.620 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-11-20T05:20:51.797 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-11-20T05:20:51.819 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-11-20T05:20:52.007 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-11-20T05:20:52.009 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-11-20T05:20:52.194 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-11-20T05:20:52.206 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-11-20T05:20:52.393 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-11-20T05:20:52.414 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-11-20T05:20:52.592 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-11-20T05:20:52.614 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-11-20T05:20:52.780 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-11-20T05:20:52.822 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-11-20T05:20:52.981 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-11-20T05:20:53.023 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-11-20T05:20:53.170 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-11-20T05:20:53.223 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-11-20T05:20:53.358 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-11-20T05:20:53.422 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-11-20T05:20:53.559 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-11-20T05:20:53.622 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-11-20T05:20:53.748 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-11-20T05:20:53.831 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-11-20T05:20:53.937 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-11-20T05:20:54.031 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-11-20T05:20:54.136 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-11-20T05:20:54.230 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-11-20T05:20:54.325 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-11-20T05:20:54.429 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-11-20T05:20:54.525 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-11-20T05:20:54.630 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-11-20T05:20:54.714 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-11-20T05:20:54.839 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-11-20T05:20:55.038 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-11-20T05:20:55.247 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-11-20T05:20:55.446 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-11-20T05:20:55.644 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-11-20T05:20:55.851 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-11-20T05:20:56.060 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-11-20T05:20:56.258 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-11-20T05:20:56.289 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2021-11-20T05:20:56.456 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-11-20T05:20:56.658 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-11-20T05:20:56.865 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-11-20T05:20:57.063 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-11-20T05:20:57.261 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-11-20T05:20:57.470 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-11-20T05:20:57.668 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-11-20T05:20:57.867 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2021-11-20T05:20:57.880 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-11-20T05:20:58.079 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-11-20T05:20:58.288 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-11-20T05:20:58.486 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-11-20T05:20:58.685 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-11-20T05:20:59.474 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2021-11-20T05:21:00.372 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2021-11-20T05:21:01.011 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2021-11-20T05:21:02.042 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2021-11-20T05:21:02.565 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2021-11-20T05:21:03.746 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2021-11-20T05:21:04.069 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2021-11-20T05:21:04.283 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-11-20T05:21:04.480 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-11-20T05:21:04.679 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-11-20T05:21:04.867 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-11-20T05:21:05.066 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-11-20T05:21:05.257 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-11-20T05:21:05.447 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2021-11-20T05:21:05.449 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-11-20T05:21:05.638 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-11-20T05:21:05.826 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-11-20T05:21:06.028 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-11-20T05:21:06.076 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:06 +0000 (0:00:15.365) 0:02:02.829 ***** 2021-11-20T05:21:06.641 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "checksum": "b3b89f3dabc35f78585fcbe6afcac330df9308f6", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "69c3df528743245bb36ff3b8fc484a02", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637385666.1743777-4161-204099436278083/source", "state": "file", "uid": 0} 2021-11-20T05:21:06.663 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:06 +0000 (0:00:00.586) 0:02:03.415 ***** 2021-11-20T05:21:06.736 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-11-20T05:21:06.738 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => { "msg": "Host smithi187.front.sepia.ceph.com is running CentOS 8 (n/a)" } 2021-11-20T05:21:06.760 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:06 +0000 (0:00:00.096) 0:02:03.512 ***** 2021-11-20T05:21:06.841 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-11-20T05:21:06.843 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-11-20T05:21:06.864 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:06 +0000 (0:00:00.104) 0:02:03.616 ***** 2021-11-20T05:21:06.935 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-11-20T05:21:06.937 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:06.959 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:06 +0000 (0:00:00.094) 0:02:03.711 ***** 2021-11-20T05:21:07.072 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** changed: [smithi058.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2021-11-20T05:21:07.271 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-11-20T05:21:07.273 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.080339", "end": "2021-11-20 05:21:07.250048", "rc": 0, "start": "2021-11-20 05:21:07.169709", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-11-20T05:21:07.293 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:07 +0000 (0:00:00.334) 0:02:04.046 ***** 2021-11-20T05:21:07.362 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-11-20T05:21:07.364 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:07.385 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:07 +0000 (0:00:00.091) 0:02:04.138 ***** 2021-11-20T05:21:07.466 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-11-20T05:21:07.468 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:07.489 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:07 +0000 (0:00:00.104) 0:02:04.242 ***** 2021-11-20T05:21:07.558 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-11-20T05:21:07.560 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:07.581 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:07 +0000 (0:00:00.091) 0:02:04.334 ***** 2021-11-20T05:21:07.656 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:07.678 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:07 +0000 (0:00:00.096) 0:02:04.430 ***** 2021-11-20T05:21:07.747 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-11-20T05:21:07.749 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:07.770 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:07 +0000 (0:00:00.091) 0:02:04.522 ***** 2021-11-20T05:21:07.838 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-11-20T05:21:07.840 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:07.860 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:07 +0000 (0:00:00.090) 0:02:04.613 ***** 2021-11-20T05:21:07.940 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-11-20T05:21:07.941 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi187.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:21:07.963 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:07 +0000 (0:00:00.102) 0:02:04.715 ***** 2021-11-20T05:21:08.033 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-11-20T05:21:08.034 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:08.056 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:08 +0000 (0:00:00.093) 0:02:04.809 ***** 2021-11-20T05:21:08.126 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-11-20T05:21:08.127 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:08.148 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:08 +0000 (0:00:00.092) 0:02:04.901 ***** 2021-11-20T05:21:08.218 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-11-20T05:21:08.220 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:08.240 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:08 +0000 (0:00:00.091) 0:02:04.993 ***** 2021-11-20T05:21:08.311 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-11-20T05:21:08.313 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:21:08.333 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:08 +0000 (0:00:00.092) 0:02:05.086 ***** 2021-11-20T05:21:08.413 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:08.437 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:08 +0000 (0:00:00.104) 0:02:05.190 ***** 2021-11-20T05:21:08.507 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-11-20T05:21:08.509 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:21:08.530 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:08 +0000 (0:00:00.092) 0:02:05.282 ***** 2021-11-20T05:21:08.599 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-11-20T05:21:08.601 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:08.625 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:08 +0000 (0:00:00.094) 0:02:05.377 ***** 2021-11-20T05:21:08.705 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-11-20T05:21:08.707 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:08.728 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:08 +0000 (0:00:00.103) 0:02:05.481 ***** 2021-11-20T05:21:08.735 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:21:08.736 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2021-11-20T05:21:08.791 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:08.813 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:08 +0000 (0:00:00.084) 0:02:05.566 ***** 2021-11-20T05:21:08.885 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-11-20T05:21:08.886 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:08.908 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:08 +0000 (0:00:00.094) 0:02:05.660 ***** 2021-11-20T05:21:08.925 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:21:08.927 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-11-20T05:21:08.969 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi187.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:21:08.990 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:08 +0000 (0:00:00.082) 0:02:05.743 ***** 2021-11-20T05:21:09.060 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:09.081 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:09 +0000 (0:00:00.091) 0:02:05.834 ***** 2021-11-20T05:21:09.140 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:21:09.142 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-11-20T05:21:09.152 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:09.175 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:09 +0000 (0:00:00.093) 0:02:05.927 ***** 2021-11-20T05:21:09.244 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:09.267 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:09 +0000 (0:00:00.091) 0:02:06.019 ***** 2021-11-20T05:21:09.337 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-11-20T05:21:09.339 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [Remove revoked users] **************************************************** ok: [smithi058.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-11-20T05:21:09.368 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:09 +0000 (0:00:00.101) 0:02:06.121 ***** 2021-11-20T05:21:09.438 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-11-20T05:21:09.440 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:09.461 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:09 +0000 (0:00:00.092) 0:02:06.213 ***** 2021-11-20T05:21:09.542 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-11-20T05:21:09.544 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [Remove revoked users] **************************************************** ok: [smithi058.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-11-20T05:21:09.565 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:09 +0000 (0:00:00.104) 0:02:06.318 ***** 2021-11-20T05:21:09.635 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi187.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:21:09.657 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:09 +0000 (0:00:00.091) 0:02:06.409 ***** 2021-11-20T05:21:09.726 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-11-20T05:21:09.728 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:09.749 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:09 +0000 (0:00:00.091) 0:02:06.501 ***** 2021-11-20T05:21:09.755 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:21:09.757 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-11-20T05:21:09.822 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:09.847 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:09 +0000 (0:00:00.098) 0:02:06.599 ***** 2021-11-20T05:21:09.938 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:21:09.939 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-11-20T05:21:10.108 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** ok: [smithi187.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:21:10.133 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:10 +0000 (0:00:00.285) 0:02:06.885 ***** 2021-11-20T05:21:10.149 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:21:10.151 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-11-20T05:21:10.349 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-11-20T05:21:10.551 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-11-20T05:21:10.752 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-11-20T05:21:10.789 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-11-20T05:21:10.790 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-11-20T05:21:10.814 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:10 +0000 (0:00:00.681) 0:02:07.566 ***** 2021-11-20T05:21:11.219 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-11-20T05:21:11.245 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:11 +0000 (0:00:00.431) 0:02:07.997 ***** 2021-11-20T05:21:11.388 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-11-20T05:21:11.390 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "checksum": "b3b89f3dabc35f78585fcbe6afcac330df9308f6", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "69c3df528743245bb36ff3b8fc484a02", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637385670.908245-4762-248860952591834/source", "state": "file", "uid": 0} 2021-11-20T05:21:11.413 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:11 +0000 (0:00:00.168) 0:02:08.166 ***** 2021-11-20T05:21:11.484 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-11-20T05:21:11.486 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => { "msg": "Host smithi058.front.sepia.ceph.com is running CentOS 8 (n/a)" } 2021-11-20T05:21:11.508 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:11 +0000 (0:00:00.094) 0:02:08.260 ***** 2021-11-20T05:21:11.578 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-11-20T05:21:11.580 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-11-20T05:21:11.602 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:11 +0000 (0:00:00.094) 0:02:08.355 ***** 2021-11-20T05:21:11.672 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-11-20T05:21:11.673 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:11.693 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:11 +0000 (0:00:00.091) 0:02:08.446 ***** 2021-11-20T05:21:11.794 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-11-20T05:21:11.795 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-11-20T05:21:12.037 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-11-20T05:21:12.039 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.105993", "end": "2021-11-20 05:21:12.021826", "rc": 0, "start": "2021-11-20 05:21:11.915833", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-11-20T05:21:12.059 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:12 +0000 (0:00:00.365) 0:02:08.812 ***** 2021-11-20T05:21:12.128 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-11-20T05:21:12.130 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:12.150 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:12 +0000 (0:00:00.090) 0:02:08.903 ***** 2021-11-20T05:21:12.222 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-11-20T05:21:12.224 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:12.245 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:12 +0000 (0:00:00.094) 0:02:08.997 ***** 2021-11-20T05:21:12.252 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-11-20T05:21:12.254 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-11-20T05:21:12.278 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:12 +0000 (0:00:00.033) 0:02:09.031 ***** 2021-11-20T05:21:12.317 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-11-20T05:21:12.318 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:12.341 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:12 +0000 (0:00:00.062) 0:02:09.093 ***** 2021-11-20T05:21:12.359 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-11-20T05:21:12.361 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:12.382 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:12 +0000 (0:00:00.041) 0:02:09.135 ***** 2021-11-20T05:21:12.411 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-11-20T05:21:12.413 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:12.435 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:12 +0000 (0:00:00.052) 0:02:09.188 ***** 2021-11-20T05:21:12.454 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:12.477 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:12 +0000 (0:00:00.041) 0:02:09.229 ***** 2021-11-20T05:21:12.505 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-11-20T05:21:12.507 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:12.530 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:12 +0000 (0:00:00.052) 0:02:09.282 ***** 2021-11-20T05:21:12.548 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-11-20T05:21:12.549 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:12.572 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:12 +0000 (0:00:00.042) 0:02:09.325 ***** 2021-11-20T05:21:12.600 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-11-20T05:21:12.602 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:12.625 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:12 +0000 (0:00:00.053) 0:02:09.378 ***** 2021-11-20T05:21:12.643 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-11-20T05:21:12.645 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:12.668 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:12 +0000 (0:00:00.042) 0:02:09.420 ***** 2021-11-20T05:21:12.922 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi058.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi058.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} Saturday 20 November 2021 05:21:12 +0000 (0:00:00.068) 0:02:09.489 ***** TASK [common : Mask sleep units] *********************************************** skipping: [smithi187.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} Saturday 20 November 2021 05:21:12 +0000 (0:00:00.075) 0:02:09.565 ***** TASK [common : Set have_entitlements] ****************************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Saturday 20 November 2021 05:21:12 +0000 (0:00:00.031) 0:02:09.596 ***** TASK [common : Get the current timezone] *************************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Saturday 20 November 2021 05:21:12 +0000 (0:0 2021-11-20T05:21:12.925 INFO:teuthology.task.ansible.out:0:00.067) 0:02:09.664 ***** 2021-11-20T05:21:12.929 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-11-20T05:21:12.931 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:12.953 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:12 +0000 (0:00:00.042) 0:02:09.706 ***** 2021-11-20T05:21:12.973 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-11-20T05:21:12.975 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:12.996 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:12 +0000 (0:00:00.042) 0:02:09.749 ***** 2021-11-20T05:21:13.025 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-11-20T05:21:13.027 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:13.049 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:13 +0000 (0:00:00.052) 0:02:09.801 ***** 2021-11-20T05:21:13.066 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:13.092 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:13 +0000 (0:00:00.042) 0:02:09.844 ***** 2021-11-20T05:21:13.120 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-11-20T05:21:13.122 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:21:13.143 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:13 +0000 (0:00:00.052) 0:02:09.896 ***** 2021-11-20T05:21:13.161 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-11-20T05:21:13.162 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:13.184 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:13 +0000 (0:00:00.040) 0:02:09.937 ***** 2021-11-20T05:21:13.214 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:13.237 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:13 +0000 (0:00:00.052) 0:02:09.990 ***** 2021-11-20T05:21:13.255 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-11-20T05:21:13.256 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:13.280 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:13 +0000 (0:00:00.042) 0:02:10.033 ***** 2021-11-20T05:21:13.318 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-11-20T05:21:13.320 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:21:13.343 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:13 +0000 (0:00:00.063) 0:02:10.096 ***** 2021-11-20T05:21:13.412 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-11-20T05:21:13.414 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:13.435 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:13 +0000 (0:00:00.092) 0:02:10.188 ***** 2021-11-20T05:21:13.516 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Set the hardware clock] ***************************************** changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.045014", "end": "2021-11-20 05:21:13.500324", "rc": 0, "start": "2021-11-20 05:21:13.455310", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:21:13.538 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:13 +0000 (0:00:00.102) 0:02:10.290 ***** 2021-11-20T05:21:13.559 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:13 +0000 (0:00:00.020) 0:02:10.311 ***** 2021-11-20T05:21:13.620 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-11-20T05:21:13.622 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:13.649 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:13 +0000 (0:00:00.089) 0:02:10.400 ***** 2021-11-20T05:21:13.730 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:13.751 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:13 +0000 (0:00:00.103) 0:02:10.504 ***** 2021-11-20T05:21:13.821 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-11-20T05:21:13.823 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:21:13.844 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:13 +0000 (0:00:00.093) 0:02:10.597 ***** 2021-11-20T05:21:13.946 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-11-20T05:21:13.948 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:13.970 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:13 +0000 (0:00:00.125) 0:02:10.722 ***** 2021-11-20T05:21:14.039 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-11-20T05:21:14.041 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:14.069 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:14 +0000 (0:00:00.099) 0:02:10.822 ***** 2021-11-20T05:21:14.141 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-11-20T05:21:14.143 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:14.164 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:14 +0000 (0:00:00.094) 0:02:10.916 ***** 2021-11-20T05:21:14.233 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-11-20T05:21:14.235 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:14.256 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:14 +0000 (0:00:00.092) 0:02:11.009 ***** 2021-11-20T05:21:14.326 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-11-20T05:21:14.328 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:14.348 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:14 +0000 (0:00:00.091) 0:02:11.100 ***** 2021-11-20T05:21:14.427 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-11-20T05:21:14.429 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:14.449 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:14 +0000 (0:00:00.100) 0:02:11.201 ***** 2021-11-20T05:21:14.528 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-11-20T05:21:14.530 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:21:14.552 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:14 +0000 (0:00:00.102) 0:02:11.303 ***** 2021-11-20T05:21:14.622 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-11-20T05:21:14.624 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:14.644 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:14 +0000 (0:00:00.092) 0:02:11.396 ***** 2021-11-20T05:21:14.725 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-11-20T05:21:14.727 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:14.748 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:14 +0000 (0:00:00.103) 0:02:11.500 ***** 2021-11-20T05:21:15.024 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-11-20T05:21:15.025 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:21:15.047 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:15 +0000 (0:00:00.298) 0:02:11.799 ***** 2021-11-20T05:21:15.605 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-11-20T05:21:15.607 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-11-20T05:21:16.070 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-11-20T05:21:16.104 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:16 +0000 (0:00:01.057) 0:02:12.856 ***** 2021-11-20T05:21:16.640 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-11-20T05:21:16.642 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-11-20T05:21:17.095 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-11-20T05:21:17.118 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:17 +0000 (0:00:01.013) 0:02:13.870 ***** 2021-11-20T05:21:17.199 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:17.218 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:17 +0000 (0:00:00.100) 0:02:13.971 ***** 2021-11-20T05:21:17.287 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-11-20T05:21:17.289 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:17.309 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:17 +0000 (0:00:00.090) 0:02:14.062 ***** 2021-11-20T05:21:17.378 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-11-20T05:21:17.380 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:17.402 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:17 +0000 (0:00:00.092) 0:02:14.154 ***** 2021-11-20T05:21:17.473 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:17.494 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:17 +0000 (0:00:00.092) 0:02:14.247 ***** 2021-11-20T05:21:17.566 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-11-20T05:21:17.568 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-11-20T05:21:17.578 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-11-20T05:21:17.580 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi058.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-11-20T05:21:17.602 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:17 +0000 (0:00:00.107) 0:02:14.354 ***** 2021-11-20T05:21:17.675 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-11-20T05:21:17.677 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:17.697 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:17 +0000 (0:00:00.095) 0:02:14.450 ***** 2021-11-20T05:21:17.767 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-11-20T05:21:17.769 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:17.789 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:17 +0000 (0:00:00.092) 0:02:14.542 ***** 2021-11-20T05:21:17.855 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:17.877 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:17 +0000 (0:00:00.087) 0:02:14.630 ***** 2021-11-20T05:21:17.947 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-11-20T05:21:17.949 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:17.973 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:17 +0000 (0:00:00.095) 0:02:14.725 ***** 2021-11-20T05:21:18.033 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:18.055 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:18 +0000 (0:00:00.081) 0:02:14.807 ***** 2021-11-20T05:21:18.250 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-11-20T05:21:18.252 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:21:18.275 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:18 +0000 (0:00:00.220) 0:02:15.028 ***** 2021-11-20T05:21:18.345 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-11-20T05:21:18.347 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:18.368 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:18 +0000 (0:00:00.093) 0:02:15.121 ***** 2021-11-20T05:21:18.438 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-11-20T05:21:18.440 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:18.461 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:18 +0000 (0:00:00.092) 0:02:15.213 ***** 2021-11-20T05:21:18.522 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-11-20T05:21:18.524 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.264494", "end": "2021-11-20 05:21:18.500403", "rc": 0, "start": "2021-11-20 05:21:18.235909", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:18.556 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:18 +0000 (0:00:00.095) 0:02:15.309 ***** 2021-11-20T05:21:18.577 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:18 +0000 (0:00:00.021) 0:02:15.330 ***** 2021-11-20T05:21:18.647 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-11-20T05:21:18.649 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:18.670 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:18 +0000 (0:00:00.093) 0:02:15.423 ***** 2021-11-20T05:21:19.177 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-11-20T05:21:19.199 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:19 +0000 (0:00:00.529) 0:02:15.952 ***** 2021-11-20T05:21:21.087 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-11-20T05:21:21.088 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:21:21.109 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:21 +0000 (0:00:01.909) 0:02:17.861 ***** 2021-11-20T05:21:21.649 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-11-20T05:21:21.651 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-11-20T05:21:22.086 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-11-20T05:21:22.513 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-11-20T05:21:22.547 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:22 +0000 (0:00:01.438) 0:02:19.300 ***** 2021-11-20T05:21:22.742 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-11-20T05:21:22.744 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:21:22.764 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:22 +0000 (0:00:00.216) 0:02:19.517 ***** 2021-11-20T05:21:23.037 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-11-20T05:21:23.039 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:21:23.062 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:23 +0000 (0:00:00.297) 0:02:19.814 ***** 2021-11-20T05:21:23.133 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-11-20T05:21:23.135 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:23.160 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:23 +0000 (0:00:00.098) 0:02:19.913 ***** 2021-11-20T05:21:23.241 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:23.266 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:23 +0000 (0:00:00.105) 0:02:20.019 ***** 2021-11-20T05:21:23.274 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-11-20T05:21:23.276 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-11-20T05:21:23.329 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:23.352 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:23 +0000 (0:00:00.086) 0:02:20.105 ***** 2021-11-20T05:21:23.420 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-11-20T05:21:23.422 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:23.442 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:23 +0000 (0:00:00.089) 0:02:20.195 ***** 2021-11-20T05:21:23.698 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-11-20T05:21:23.704 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-11-20T05:21:23.971 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-11-20T05:21:23.973 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-11-20T05:21:23.993 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:23 +0000 (0:00:00.551) 0:02:20.746 ***** 2021-11-20T05:21:24.179 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-11-20T05:21:24.181 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637385683.7220273-6552-264836860785382/source", "state": "file", "uid": 0} 2021-11-20T05:21:24.245 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:24 +0000 (0:00:00.251) 0:02:20.997 ***** 2021-11-20T05:21:24.283 INFO:teuthology.task.ansible.out: TASK [common : Include dnf_system vars] **************************************** 2021-11-20T05:21:24.285 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/dnf_systems.yml"], "changed": false} 2021-11-20T05:21:24.306 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:24 +0000 (0:00:00.061) 0:02:21.059 ***** 2021-11-20T05:21:24.501 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-11-20T05:21:24.503 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-11-20T05:21:24.538 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:24 +0000 (0:00:00.231) 0:02:21.291 ***** 2021-11-20T05:21:24.587 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-11-20T05:21:24.589 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T05:21:24.599 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T05:21:24.610 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T05:21:24.634 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:24 +0000 (0:00:00.095) 0:02:21.386 ***** 2021-11-20T05:21:25.967 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-11-20T05:21:25.969 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:21:25.991 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:25 +0000 (0:00:01.357) 0:02:22.744 ***** 2021-11-20T05:21:26.494 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-11-20T05:21:26.495 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:21:26.497 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-11-20T05:21:26.497 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-11-20T05:21:26.522 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:26 +0000 (0:00:00.530) 0:02:23.275 ***** 2021-11-20T05:21:26.555 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (non-RHEL/CentOS)] ******** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:26.574 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:26 +0000 (0:00:00.051) 0:02:23.326 ***** 2021-11-20T05:21:26.962 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi058.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-11-20T05:21:27.047 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-11-20T05:21:27.068 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:27 +0000 (0:00:00.494) 0:02:23.821 ***** 2021-11-20T05:21:27.435 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-11-20T05:21:27.437 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-11-20T05:21:27.463 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:27 +0000 (0:00:00.394) 0:02:24.216 ***** 2021-11-20T05:21:27.545 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-11-20T05:21:27.548 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-11-20T05:21:27.578 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:27 +0000 (0:00:00.114) 0:02:24.330 ***** 2021-11-20T05:21:27.620 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-11-20T05:21:27.622 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-11-20T05:21:27.632 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-11-20T05:21:27.655 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:27 +0000 (0:00:00.077) 0:02:24.407 ***** 2021-11-20T05:21:27.675 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:21:27.700 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:27 +0000 (0:00:00.045) 0:02:24.453 ***** 2021-11-20T05:21:27.870 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [smithi187.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:21:27.894 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:27 +0000 (0:00:00.193) 0:02:24.646 ***** 2021-11-20T05:21:28.120 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-11-20T05:21:28.122 INFO:teuthology.task.ansible.out:fatal: [smithi187.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-11-20T05:21:28.143 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:28 +0000 (0:00:00.248) 0:02:24.895 ***** 2021-11-20T05:21:28.181 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-11-20T05:21:28.183 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:28.207 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:28 +0000 (0:00:00.064) 0:02:24.960 ***** 2021-11-20T05:21:28.215 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-11-20T05:21:28.217 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-11-20T05:21:28.629 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-11-20T05:21:28.682 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-11-20T05:21:28.684 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-11-20T05:21:28.706 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:28 +0000 (0:00:00.498) 0:02:25.458 ***** 2021-11-20T05:21:29.141 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-11-20T05:21:29.144 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637385688.6566715-7273-193954992421589/source", "state": "file", "uid": 0} 2021-11-20T05:21:29.165 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:29 +0000 (0:00:00.459) 0:02:25.918 ***** 2021-11-20T05:21:29.203 INFO:teuthology.task.ansible.out: TASK [common : Include dnf_system vars] **************************************** 2021-11-20T05:21:29.205 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/dnf_systems.yml"], "changed": false} 2021-11-20T05:21:29.226 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:29 +0000 (0:00:00.060) 0:02:25.979 ***** 2021-11-20T05:21:29.359 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-11-20T05:21:29.360 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-11-20 05:18:14 UTC", "ActiveEnterTimestampMonotonic": "19454116", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target systemd-tmpfiles-setup.service systemd-journald.socket system.slice local-fs.target tmp.mount time-sync.target sysinit.target -.mount var-run.mount network-online.target basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-11-20 05:18:14 UTC", "AssertTimestampMonotonic": "19429852", "Before": "multi-user.target xdm.service getty@tty1.service shutdown.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-11-20 05:18:14 UTC", "Co 2021-11-20T05:21:29.362 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "19429852", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1266", "ExecMainStartTimestamp": "Sat 2021-11-20 05:18:14 UTC", "ExecMainStartTimestampMonotonic": "19453250", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Sat 2021-11-20 05:18:14 UTC] ; stop_time=[n/a] ; pid=1266 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-11-20 05:18:14 UTC", "InactiveExitTimestampMonotonic": "194541 2021-11-20T05:21:29.363 INFO:teuthology.task.ansible.out:16", "InvocationID": "0d7161cc98ec4c18a58e74d615735f6e", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1266", "MemoryAccounting": "yes", "MemoryCurrent": "671744", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectC 2021-11-20T05:21:29.365 INFO:teuthology.task.ansible.out:ontrolGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-11-20 05:18:14 UTC", "StateChangeTimestampMonotonic": "19454116", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-11-20 05:18:14 UTC", "WatchdogTimestampMonotonic": "19454095", "WatchdogUSec": "0"}} 2021-11-20T05:21:29.384 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:29 +0000 (0:00:00.157) 0:02:26.136 ***** 2021-11-20T05:21:29.579 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-11-20T05:21:29.580 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001776", "end": "2021-11-20 05:21:29.555838", "rc": 0, "start": "2021-11-20 05:21:29.554062", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-11-20T05:21:29.601 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:29 +0000 (0:00:00.217) 0:02:26.354 ***** 2021-11-20T05:21:30.435 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-11-20T05:21:30.437 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-11-20T05:21:30.458 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:30 +0000 (0:00:00.856) 0:02:27.210 ***** 2021-11-20T05:21:30.496 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-11-20T05:21:30.497 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T05:21:30.507 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T05:21:30.519 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T05:21:30.529 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T05:21:30.552 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:30 +0000 (0:00:00.094) 0:02:27.305 ***** 2021-11-20T05:21:31.468 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-11-20T05:21:31.470 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:21:31.493 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:31 +0000 (0:00:00.940) 0:02:28.245 ***** 2021-11-20T05:21:32.486 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-11-20T05:21:32.488 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:21:32.512 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:32 +0000 (0:00:01.018) 0:02:29.264 ***** 2021-11-20T05:21:32.540 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (non-RHEL/CentOS)] ******** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:32.562 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:32 +0000 (0:00:00.050) 0:02:29.315 ***** 2021-11-20T05:21:33.044 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-11-20T05:21:33.046 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-11-20T05:21:33.069 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:33 +0000 (0:00:00.506) 0:02:29.821 ***** 2021-11-20T05:21:33.546 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-11-20T05:21:33.548 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-11-20T05:21:33.569 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:33 +0000 (0:00:00.499) 0:02:30.321 ***** 2021-11-20T05:21:33.606 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-11-20T05:21:33.608 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-11-20T05:21:33.618 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-11-20T05:21:33.640 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:33 +0000 (0:00:00.071) 0:02:30.392 ***** 2021-11-20T05:21:33.854 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-11-20T05:21:33.856 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:21:33.876 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:33 +0000 (0:00:00.236) 0:02:30.628 ***** 2021-11-20T05:21:34.122 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** fatal: [smithi058.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-11-20T05:21:34.144 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:34 +0000 (0:00:00.267) 0:02:30.896 ***** 2021-11-20T05:21:34.171 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-11-20T05:21:34.173 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:21:34.193 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:34 +0000 (0:00:00.049) 0:02:30.946 ***** 2021-11-20T05:21:34.680 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-11-20T05:21:34.703 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:34 +0000 (0:00:00.509) 0:02:31.455 ***** 2021-11-20T05:21:35.024 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-11-20T05:21:35.025 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-11-20 05:18:09 UTC", "ActiveEnterTimestampMonotonic": "22917624", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket time-sync.target basic.target sysinit.target network-online.target -.mount nss-lookup.target systemd-tmpfiles-setup.service tmp.mount var-run.mount local-fs.target system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-11-20 05:18:09 UTC", "AssertTimestampMonotonic": "22906472", "Before": "getty@tty1.service xdm.service shutdown.target plymouth-quit.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-11-20 05:18:09 UTC", "Co 2021-11-20T05:21:35.026 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "22906472", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1274", "ExecMainStartTimestamp": "Sat 2021-11-20 05:18:09 UTC", "ExecMainStartTimestampMonotonic": "22917538", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Sat 2021-11-20 05:18:09 UTC] ; stop_time=[n/a] ; pid=1274 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-11-20 05:18:09 UTC", "InactiveExitTimestampMonotonic": "229176 2021-11-20T05:21:35.028 INFO:teuthology.task.ansible.out:24", "InvocationID": "c70898301b7b420da3ebfa6cdd8456ff", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127267", "LimitNPROCSoft": "127267", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127267", "LimitSIGPENDINGSoft": "127267", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1274", "MemoryAccounting": "yes", "MemoryCurrent": "667648", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectC 2021-11-20T05:21:35.030 INFO:teuthology.task.ansible.out:ontrolGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-11-20 05:18:09 UTC", "StateChangeTimestampMonotonic": "22917624", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203627", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-11-20 05:18:09 UTC", "WatchdogTimestampMonotonic": "22917623", "WatchdogUSec": "0"}} 2021-11-20T05:21:35.047 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:35 +0000 (0:00:00.344) 0:02:31.800 ***** 2021-11-20T05:21:35.245 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-11-20T05:21:35.247 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001852", "end": "2021-11-20 05:21:35.227402", "rc": 0, "start": "2021-11-20 05:21:35.225550", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-11-20T05:21:35.267 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:35 +0000 (0:00:00.220) 0:02:32.020 ***** 2021-11-20T05:21:37.189 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-11-20T05:21:37.190 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:21:37.212 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:21:37 +0000 (0:00:01.944) 0:02:33.964 ***** 2021-11-20T05:22:11.646 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-11-20T05:22:11.648 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-83.el8.noarch", "Installed: selinux-policy-targeted-3.14.3-83.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2021-11-20T05:22:11.670 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:11 +0000 (0:00:34.458) 0:03:08.422 ***** 2021-11-20T05:22:12.376 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-11-20T05:22:12.378 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-11-20T05:22:12.398 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:12 +0000 (0:00:00.728) 0:03:09.151 ***** 2021-11-20T05:22:18.068 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-11-20T05:22:18.070 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-83.el8.noarch", "Installed: selinux-policy-targeted-3.14.3-83.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2021-11-20T05:22:18.092 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:18 +0000 (0:00:05.693) 0:03:14.845 ***** 2021-11-20T05:22:18.570 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-11-20T05:22:18.571 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-11-20T05:22:18.592 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:18 +0000 (0:00:00.499) 0:03:15.345 ***** 2021-11-20T05:22:28.358 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-11-20T05:22:28.360 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:15.756709", "end": "2021-11-20 05:22:28.330839", "failed_when_result": false, "rc": 0, "start": "2021-11-20 05:22:12.574130", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-11-20T05:22:28.387 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:28 +0000 (0:00:09.795) 0:03:25.140 ***** 2021-11-20T05:22:28.846 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-11-20T05:22:28.848 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-11-20T05:22:28.869 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:28 +0000 (0:00:00.481) 0:03:25.622 ***** 2021-11-20T05:22:29.074 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-11-20T05:22:29.076 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.015334", "end": "2021-11-20 05:22:29.052937", "rc": 0, "start": "2021-11-20 05:22:29.037603", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:22:29.105 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:29 +0000 (0:00:00.235) 0:03:25.858 ***** 2021-11-20T05:22:29.303 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-11-20T05:22:29.305 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.010572", "end": "2021-11-20 05:22:29.287983", "rc": 0, "start": "2021-11-20 05:22:29.277411", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:22:29.326 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:29 +0000 (0:00:00.221) 0:03:26.079 ***** 2021-11-20T05:22:35.118 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-11-20T05:22:35.120 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.312520", "end": "2021-11-20 05:22:35.090783", "failed_when_result": false, "rc": 0, "start": "2021-11-20 05:22:18.778263", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-11-20T05:22:35.142 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:35 +0000 (0:00:05.815) 0:03:31.895 ***** 2021-11-20T05:22:35.618 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-11-20T05:22:35.621 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-11-20T05:22:35.641 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:35 +0000 (0:00:00.498) 0:03:32.393 ***** 2021-11-20T05:22:35.855 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-11-20T05:22:35.857 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.016202", "end": "2021-11-20 05:22:35.834609", "rc": 0, "start": "2021-11-20 05:22:35.818407", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:22:35.877 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:35 +0000 (0:00:00.236) 0:03:32.630 ***** 2021-11-20T05:22:36.081 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-11-20T05:22:36.083 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.014458", "end": "2021-11-20 05:22:36.065637", "rc": 0, "start": "2021-11-20 05:22:36.051179", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:22:36.103 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:36 +0000 (0:00:00.225) 0:03:32.856 ***** 2021-11-20T05:22:45.134 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-11-20T05:22:45.136 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.601302", "end": "2021-11-20 05:22:45.108346", "rc": 0, "start": "2021-11-20 05:22:29.507044", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:22:45.165 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:45 +0000 (0:00:09.061) 0:03:41.917 ***** 2021-11-20T05:22:45.373 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-11-20T05:22:45.374 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-11-20T05:22:45.396 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:45 +0000 (0:00:00.230) 0:03:42.148 ***** 2021-11-20T05:22:45.424 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-11-20T05:22:45.426 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-11-20T05:22:45.447 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:45 +0000 (0:00:00.051) 0:03:42.200 ***** 2021-11-20T05:22:45.490 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-11-20T05:22:45.492 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-11-20T05:22:45.513 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:45 +0000 (0:00:00.065) 0:03:42.265 ***** 2021-11-20T05:22:45.551 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-11-20T05:22:45.553 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-20T05:22:45.555 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-20T05:22:45.576 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:45 +0000 (0:00:00.063) 0:03:42.329 ***** 2021-11-20T05:22:45.614 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-11-20T05:22:45.616 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-20T05:22:45.617 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-20T05:22:45.638 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:45 +0000 (0:00:00.061) 0:03:42.390 ***** 2021-11-20T05:22:45.824 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-11-20T05:22:45.826 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-20T05:22:45.846 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:45 +0000 (0:00:00.208) 0:03:42.598 ***** 2021-11-20T05:22:46.041 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-11-20T05:22:46.043 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-11-20T05:22:46.065 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:46 +0000 (0:00:00.218) 0:03:42.817 ***** 2021-11-20T05:22:46.094 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-11-20T05:22:46.096 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:22:46.125 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:46 +0000 (0:00:00.060) 0:03:42.878 ***** 2021-11-20T05:22:46.156 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-11-20T05:22:46.158 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:22:46.178 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:46 +0000 (0:00:00.052) 0:03:42.931 ***** 2021-11-20T05:22:46.404 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-11-20T05:22:46.406 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-11-20T05:22:46.426 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:46 +0000 (0:00:00.247) 0:03:43.178 ***** 2021-11-20T05:22:46.983 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-11-20T05:22:46.985 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-11-20T05:22:47.007 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:47 +0000 (0:00:00.581) 0:03:43.760 ***** 2021-11-20T05:22:47.034 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-11-20T05:22:47.036 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi187"}, "changed": false} 2021-11-20T05:22:47.055 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:47 +0000 (0:00:00.048) 0:03:43.808 ***** 2021-11-20T05:22:47.834 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-11-20T05:22:47.836 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi187.front.sepia.ceph.com", "ansible_hostname": "smithi187", "ansible_nodename": "smithi187"}, "changed": true, "name": "smithi187"} 2021-11-20T05:22:47.866 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:47 +0000 (0:00:00.810) 0:03:44.619 ***** 2021-11-20T05:22:48.317 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-11-20T05:22:48.319 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-11-20T05:22:48.339 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:48 +0000 (0:00:00.473) 0:03:45.092 ***** 2021-11-20T05:22:48.804 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-11-20T05:22:48.806 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-11-20T05:22:48.836 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:48 +0000 (0:00:00.496) 0:03:45.588 ***** 2021-11-20T05:22:49.354 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-11-20T05:22:49.355 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-11-20T05:22:49.376 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:49 +0000 (0:00:00.540) 0:03:46.129 ***** 2021-11-20T05:22:49.844 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-11-20T05:22:49.846 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-11-20T05:22:49.869 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:49 +0000 (0:00:00.492) 0:03:46.621 ***** 2021-11-20T05:22:50.074 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-11-20T05:22:50.076 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-20T05:22:50.098 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:50 +0000 (0:00:00.229) 0:03:46.851 ***** 2021-11-20T05:22:50.293 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-11-20T05:22:50.295 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-11-20T05:22:50.316 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:50 +0000 (0:00:00.217) 0:03:47.068 ***** 2021-11-20T05:22:50.541 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-11-20T05:22:50.543 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-11-20T05:22:50.563 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:50 +0000 (0:00:00.247) 0:03:47.316 ***** 2021-11-20T05:22:51.036 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-11-20T05:22:51.037 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-11-20T05:22:51.057 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:51 +0000 (0:00:00.494) 0:03:47.810 ***** 2021-11-20T05:22:51.520 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-11-20T05:22:51.522 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-11-20T05:22:51.547 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:51 +0000 (0:00:00.489) 0:03:48.300 ***** 2021-11-20T05:22:52.425 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-11-20T05:22:52.427 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.120944", "end": "2021-11-20 05:22:52.400494", "rc": 0, "start": "2021-11-20 05:22:36.279550", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:22:52.448 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:52 +0000 (0:00:00.901) 0:03:49.201 ***** 2021-11-20T05:22:52.653 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-11-20T05:22:52.655 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-11-20T05:22:52.675 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:52 +0000 (0:00:00.227) 0:03:49.428 ***** 2021-11-20T05:22:52.713 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-11-20T05:22:52.715 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-11-20T05:22:52.736 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:52 +0000 (0:00:00.060) 0:03:49.488 ***** 2021-11-20T05:22:52.773 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-11-20T05:22:52.775 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-11-20T05:22:52.801 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:52 +0000 (0:00:00.065) 0:03:49.553 ***** 2021-11-20T05:22:52.840 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-11-20T05:22:52.842 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-20T05:22:52.844 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-20T05:22:52.874 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:52 +0000 (0:00:00.073) 0:03:49.627 ***** 2021-11-20T05:22:52.922 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-11-20T05:22:52.924 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-20T05:22:52.926 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-20T05:22:52.946 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:52 +0000 (0:00:00.072) 0:03:49.699 ***** 2021-11-20T05:22:53.139 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-11-20T05:22:53.141 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-20T05:22:53.161 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:53 +0000 (0:00:00.214) 0:03:49.914 ***** 2021-11-20T05:22:53.355 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-11-20T05:22:53.357 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-11-20T05:22:53.377 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:53 +0000 (0:00:00.215) 0:03:50.130 ***** 2021-11-20T05:22:53.404 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-11-20T05:22:53.405 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:22:53.426 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:53 +0000 (0:00:00.049) 0:03:50.179 ***** 2021-11-20T05:22:53.454 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-11-20T05:22:53.456 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:22:53.476 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:53 +0000 (0:00:00.049) 0:03:50.228 ***** 2021-11-20T05:22:53.705 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-11-20T05:22:53.706 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-11-20T05:22:53.727 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:53 +0000 (0:00:00.251) 0:03:50.479 ***** 2021-11-20T05:22:54.160 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-11-20T05:22:54.162 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.424121", "end": "2021-11-20 05:22:54.138885", "rc": 0, "start": "2021-11-20 05:22:51.714764", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:22:54.183 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:54 +0000 (0:00:00.456) 0:03:50.936 ***** 2021-11-20T05:22:54.295 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-11-20T05:22:54.296 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-11-20T05:22:54.319 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:54 +0000 (0:00:00.135) 0:03:51.071 ***** 2021-11-20T05:22:54.357 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-11-20T05:22:54.359 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi058"}, "changed": false} 2021-11-20T05:22:54.379 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:54 +0000 (0:00:00.060) 0:03:51.132 ***** 2021-11-20T05:22:54.397 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-11-20T05:22:54.400 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the file module with state=absent rather than running 'rm'. If you need to use command because file is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rvf /etc/yum.repos.d/*-Stream-*", "delta": "0:00:00.002641", "end": "2021-11-20 05:22:54.373705", "rc": 0, "start": "2021-11-20 05:22:54.371064", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:22:54.425 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:54 +0000 (0:00:00.045) 0:03:51.177 ***** 2021-11-20T05:22:54.463 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:22:54.465 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"dash_stream": "-stream"}, "changed": false} 2021-11-20T05:22:54.495 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:54 +0000 (0:00:00.070) 0:03:51.248 ***** 2021-11-20T05:22:54.556 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-11-20T05:22:54.557 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml) => {"ansible_facts": {"packages_to_upgrade": ["systemd"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"} 2021-11-20T05:22:54.578 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:54 +0000 (0:00:00.082) 0:03:51.331 ***** 2021-11-20T05:22:54.824 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-11-20T05:22:54.826 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi058.front.sepia.ceph.com", "ansible_hostname": "smithi058", "ansible_nodename": "smithi058"}, "changed": true, "name": "smithi058"} 2021-11-20T05:22:54.847 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:54 +0000 (0:00:00.269) 0:03:51.600 ***** 2021-11-20T05:22:55.064 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-11-20T05:22:55.067 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "be4ef7d7f1f4743a5048cab6a4dced5e5d1e5b14", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2021-11-20T05:22:55.316 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-11-20T05:22:55.318 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-11-20T05:22:55.339 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:55 +0000 (0:00:00.491) 0:03:52.092 ***** 2021-11-20T05:22:55.505 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-11-20T05:22:55.506 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "5a66a2928a1d64d8d11230f172eb6b696c57c58c", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2021-11-20T05:22:55.809 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-11-20T05:22:55.811 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-11-20T05:22:55.832 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:55 +0000 (0:00:00.492) 0:03:52.584 ***** 2021-11-20T05:22:55.943 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-11-20T05:22:55.945 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "12b6934a87ad915f812630b0a7e54663770cd059", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2021-11-20T05:22:56.406 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-11-20T05:22:56.408 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-11-20T05:22:56.429 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:56 +0000 (0:00:00.597) 0:03:53.182 ***** 2021-11-20T05:22:56.438 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** changed: [smithi187.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "c36ac65b18cdf8071cdb7bd4e46c5639be96cbac", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "md5sum": "76ff786610d5a5d6d02f2e8fd5311b89", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637385775.9689863-14371-16588688083881/source", "state": "file", "uid": 0} 2021-11-20T05:22:56.465 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:56 +0000 (0:00:00.035) 0:03:53.218 ***** 2021-11-20T05:22:56.927 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-11-20T05:22:56.929 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-11-20T05:22:56.952 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:56 +0000 (0:00:00.486) 0:03:53.704 ***** 2021-11-20T05:22:56.961 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-11-20T05:22:56.962 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-11-20T05:22:57.153 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-11-20T05:22:57.154 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-20T05:22:57.180 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:57 +0000 (0:00:00.228) 0:03:53.932 ***** 2021-11-20T05:22:57.378 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-11-20T05:22:57.381 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-11-20T05:22:57.401 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:57 +0000 (0:00:00.221) 0:03:54.153 ***** 2021-11-20T05:22:57.441 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-11-20T05:22:57.442 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'apache-arrow', 'value': {'name': 'Apache Arrow', 'baseurl': 'https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch', 'enabled': 1, 'gpgcheck': 1}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "86b924db6e383ddad921a207c09a708e6cca6a67", "dest": "/etc/yum.repos.d/apache-arrow.repo", "gid": 0, "group": "root", "item": {"key": "apache-arrow", "value": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}}, "md5sum": "183d0889f7213a1e38c7f48965d7549e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 215, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637385776.982263-14510-218714567454934/source", "state": "file", "uid": 0} 2021-11-20T05:22:57.468 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:57 +0000 (0:00:00.067) 0:03:54.221 ***** 2021-11-20T05:22:57.632 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* ok: [smithi058.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-11-20T05:22:57.655 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:57 +0000 (0:00:00.187) 0:03:54.408 ***** 2021-11-20T05:22:57.959 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** ok: [smithi187.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-11-20T05:22:58.137 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-11-20T05:22:58.138 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-11-20T05:22:58.160 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:58 +0000 (0:00:00.504) 0:03:54.912 ***** 2021-11-20T05:22:58.396 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-11-20T05:22:58.398 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-11-20T05:22:58.648 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-11-20T05:22:58.650 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-11-20T05:22:58.670 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:58 +0000 (0:00:00.510) 0:03:55.423 ***** 2021-11-20T05:22:58.835 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-11-20T05:22:58.837 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-11-20T05:22:59.316 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "md5sum": "d96965513d5dae48e1f179ee52900b67", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637385778.8617563-14629-100133037576311/source", "state": "file", "uid": 0} 2021-11-20T05:22:59.342 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:59 +0000 (0:00:00.671) 0:03:56.095 ***** 2021-11-20T05:22:59.800 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** changed: [smithi187.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.252604", "end": "2021-11-20 05:22:59.777647", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-11-20 05:22:59.525043", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2021-11-20T05:22:59.801 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2021-11-20T05:22:59.827 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:22:59 +0000 (0:00:00.484) 0:03:56.579 ***** 2021-11-20T05:23:01.502 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-11-20T05:23:01.504 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.625776", "end": "2021-11-20 05:23:01.478835", "rc": 0, "start": "2021-11-20 05:22:58.853059", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:23:01.527 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:01 +0000 (0:00:01.699) 0:03:58.279 ***** 2021-11-20T05:23:01.754 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-11-20T05:23:01.756 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rvf /etc/yum.repos.d/*-Stream-*", "delta": "0:00:00.002928", "end": "2021-11-20 05:23:01.730008", "rc": 0, "start": "2021-11-20 05:23:01.727080", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:23:01.778 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:01 +0000 (0:00:00.249) 0:03:58.529 ***** 2021-11-20T05:23:01.814 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi058.front.sepia.ceph.com] => {"ansible_facts": {"dash_stream": "-stream"}, "changed": false} 2021-11-20T05:23:01.837 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:01 +0000 (0:00:00.060) 0:03:58.589 ***** 2021-11-20T05:23:01.896 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-11-20T05:23:01.897 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml) => {"ansible_facts": {"packages_to_upgrade": ["systemd"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"} 2021-11-20T05:23:01.918 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:01 +0000 (0:00:00.081) 0:03:58.671 ***** 2021-11-20T05:23:02.408 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-11-20T05:23:02.410 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "be4ef7d7f1f4743a5048cab6a4dced5e5d1e5b14", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2021-11-20T05:23:02.869 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "5a66a2928a1d64d8d11230f172eb6b696c57c58c", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2021-11-20T05:23:03.331 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "12b6934a87ad915f812630b0a7e54663770cd059", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2021-11-20T05:23:03.832 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "c36ac65b18cdf8071cdb7bd4e46c5639be96cbac", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "md5sum": "76ff786610d5a5d6d02f2e8fd5311b89", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637385783.35726-15019-2768217825039/source", "state": "file", "uid": 0} 2021-11-20T05:23:03.856 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:03 +0000 (0:00:01.938) 0:04:00.609 ***** 2021-11-20T05:23:04.365 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-11-20T05:23:04.366 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-11-20T05:23:04.864 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'apache-arrow', 'value': {'name': 'Apache Arrow', 'baseurl': 'https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch', 'enabled': 1, 'gpgcheck': 1}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "86b924db6e383ddad921a207c09a708e6cca6a67", "dest": "/etc/yum.repos.d/apache-arrow.repo", "gid": 0, "group": "root", "item": {"key": "apache-arrow", "value": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}}, "md5sum": "183d0889f7213a1e38c7f48965d7549e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 215, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637385784.3889089-15201-217205897437553/source", "state": "file", "uid": 0} 2021-11-20T05:23:04.888 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:04 +0000 (0:00:01.031) 0:04:01.640 ***** 2021-11-20T05:23:05.383 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-11-20T05:23:05.385 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-11-20T05:23:05.839 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-11-20T05:23:06.303 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-11-20T05:23:06.808 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "md5sum": "d96965513d5dae48e1f179ee52900b67", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637385786.3347151-15288-135205683610810/source", "state": "file", "uid": 0} 2021-11-20T05:23:06.834 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:06 +0000 (0:00:01.945) 0:04:03.586 ***** 2021-11-20T05:23:07.292 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-11-20T05:23:07.293 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:23:07.315 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:07 +0000 (0:00:00.481) 0:04:04.068 ***** 2021-11-20T05:23:07.323 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-11-20T05:23:07.325 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.266986", "end": "2021-11-20 05:23:07.286581", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-11-20 05:23:07.019595", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2021-11-20T05:23:07.351 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:07 +0000 (0:00:00.034) 0:04:04.103 ***** 2021-11-20T05:23:08.110 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-11-20T05:23:08.112 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/release.asc"} 2021-11-20T05:23:08.571 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/autobuild.asc"} 2021-11-20T05:23:09.901 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item=https://dist.apache.org/repos/dist/dev/arrow/KEYS) => {"ansible_loop_var": "item", "changed": true, "item": "https://dist.apache.org/repos/dist/dev/arrow/KEYS"} 2021-11-20T05:23:09.925 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:09 +0000 (0:00:02.575) 0:04:06.678 ***** 2021-11-20T05:23:10.306 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** [WARNING]: Consider using the yum module rather than running 'yum'. If you need to use command because yum is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.184233", "end": "2021-11-20 05:23:10.286363", "rc": 0, "start": "2021-11-20 05:23:10.102130", "stderr": "", "stderr_lines": [], "stdout": "57 files removed", "stdout_lines": ["57 files removed"]} 2021-11-20T05:23:10.327 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:10 +0000 (0:00:00.401) 0:04:07.080 ***** 2021-11-20T05:23:10.365 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:23:10.367 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-11-20T05:23:10.389 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:10 +0000 (0:00:00.061) 0:04:07.141 ***** 2021-11-20T05:23:10.427 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-11-20T05:23:10.429 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:10.449 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:10 +0000 (0:00:00.060) 0:04:07.202 ***** 2021-11-20T05:23:10.487 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:10.511 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:10 +0000 (0:00:00.060) 0:04:07.262 ***** 2021-11-20T05:23:10.715 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-11-20T05:23:10.717 INFO:teuthology.task.ansible.out:fatal: [smithi187.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007355", "end": "2021-11-20 05:23:10.691144", "msg": "non-zero return code", "rc": 1, "start": "2021-11-20 05:23:10.683789", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-11-20T05:23:10.738 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:10 +0000 (0:00:00.228) 0:04:07.491 ***** 2021-11-20T05:23:10.767 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-11-20T05:23:10.769 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:10.790 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:10 +0000 (0:00:00.051) 0:04:07.542 ***** 2021-11-20T05:23:15.122 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-11-20T05:23:15.124 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:23:15.146 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:15 +0000 (0:00:04.356) 0:04:11.899 ***** 2021-11-20T05:23:15.718 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-11-20T05:23:15.720 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/release.asc"} 2021-11-20T05:23:16.210 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/autobuild.asc"} 2021-11-20T05:23:17.732 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item=https://dist.apache.org/repos/dist/dev/arrow/KEYS) => {"ansible_loop_var": "item", "changed": true, "item": "https://dist.apache.org/repos/dist/dev/arrow/KEYS"} 2021-11-20T05:23:17.755 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:17 +0000 (0:00:02.608) 0:04:14.508 ***** 2021-11-20T05:23:18.171 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-11-20T05:23:18.173 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.196645", "end": "2021-11-20 05:23:18.144654", "rc": 0, "start": "2021-11-20 05:23:17.948009", "stderr": "", "stderr_lines": [], "stdout": "57 files removed", "stdout_lines": ["57 files removed"]} 2021-11-20T05:23:18.195 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:18 +0000 (0:00:00.439) 0:04:14.947 ***** 2021-11-20T05:23:18.233 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:23:18.235 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-11-20T05:23:18.256 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:18 +0000 (0:00:00.061) 0:04:15.009 ***** 2021-11-20T05:23:18.295 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:18.317 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:18 +0000 (0:00:00.061) 0:04:15.070 ***** 2021-11-20T05:23:18.356 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-11-20T05:23:18.358 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:18.379 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:18 +0000 (0:00:00.062) 0:04:15.132 ***** 2021-11-20T05:23:18.586 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-11-20T05:23:18.590 INFO:teuthology.task.ansible.out:fatal: [smithi058.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007918", "end": "2021-11-20 05:23:18.566783", "msg": "non-zero return code", "rc": 1, "start": "2021-11-20 05:23:18.558865", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-11-20T05:23:18.610 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:18 +0000 (0:00:00.230) 0:04:15.362 ***** 2021-11-20T05:23:18.650 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-11-20T05:23:18.652 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:18.680 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:18 +0000 (0:00:00.069) 0:04:15.432 ***** 2021-11-20T05:23:28.928 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-11-20T05:23:28.930 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2021-11-20T05:23:28.952 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:28 +0000 (0:00:10.271) 0:04:25.704 ***** 2021-11-20T05:23:32.316 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-11-20T05:23:32.318 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-11-20T05:23:32.341 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:32 +0000 (0:00:03.388) 0:04:29.093 ***** 2021-11-20T05:23:37.426 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-11-20T05:23:37.428 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2021-11-20T05:23:37.453 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:37 +0000 (0:00:05.112) 0:04:34.206 ***** 2021-11-20T05:23:40.803 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-11-20T05:23:40.806 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-11-20T05:23:40.827 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:40 +0000 (0:00:03.373) 0:04:37.579 ***** 2021-11-20T05:23:41.761 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T05:23:41.762 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: qemu-kvm-common-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: python3-dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: qemu-kvm-core-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: yum-utils-4.0.21-3.el8.noarch", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: python3-dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: yum-utils-4.0.18-4.el8.noarch"]} 2021-11-20T05:23:41.785 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:41 +0000 (0:00:00.958) 0:04:38.538 ***** 2021-11-20T05:23:41.824 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:41.846 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:41 +0000 (0:00:00.061) 0:04:38.599 ***** 2021-11-20T05:23:44.092 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-11-20T05:23:44.094 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:23:44.115 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:44 +0000 (0:00:02.268) 0:04:40.868 ***** 2021-11-20T05:23:50.364 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T05:23:50.366 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: qemu-kvm-common-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: python3-dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: qemu-kvm-core-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: yum-utils-4.0.21-3.el8.noarch", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: python3-dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: yum-utils-4.0.18-4.el8.noarch"]} 2021-11-20T05:23:50.388 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:50 +0000 (0:00:06.273) 0:04:47.141 ***** 2021-11-20T05:23:50.417 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-11-20T05:23:50.419 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:50.439 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:50 +0000 (0:00:00.050) 0:04:47.192 ***** 2021-11-20T05:23:52.476 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-20T05:23:52.478 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: systemd-pam-239-51.el8.x86_64", "Installed: systemd-239-51.el8.x86_64", "Installed: systemd-libs-239-51.el8.x86_64", "Installed: systemd-udev-239-51.el8.x86_64", "Removed: systemd-239-45.el8.x86_64", "Removed: systemd-libs-239-45.el8.x86_64", "Removed: systemd-pam-239-45.el8.x86_64", "Removed: systemd-udev-239-45.el8.x86_64"]} 2021-11-20T05:23:52.500 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:52 +0000 (0:00:02.060) 0:04:49.252 ***** 2021-11-20T05:23:52.663 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-11-20T05:23:52.665 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:23:52.687 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:52 +0000 (0:00:00.186) 0:04:49.439 ***** 2021-11-20T05:23:52.802 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-11-20T05:23:52.803 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice basic.target dbus.service sysinit.target polkit.service dbus.socket", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ebtables.service shutdown.target ipset.service iptables.service nftables.service ip6tables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate" 2021-11-20T05:23:52.805 INFO:teuthology.task.ansible.out:: "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": 2021-11-20T05:23:52.806 INFO:teuthology.task.ansible.out:"0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket system.slice sysinit.target", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "nul 2021-11-20T05:23:52.808 INFO:teuthology.task.ansible.out:l", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-11-20T05:23:52.829 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:52 +0000 (0:00:00.141) 0:04:49.581 ***** 2021-11-20T05:23:53.117 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-11-20T05:23:53.119 INFO:teuthology.task.ansible.out:fatal: [smithi187.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-11-20T05:23:53.139 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:53 +0000 (0:00:00.310) 0:04:49.892 ***** 2021-11-20T05:23:53.755 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-11-20T05:23:53.757 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-11-20T05:23:53.777 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:53 +0000 (0:00:00.637) 0:04:50.529 ***** 2021-11-20T05:23:53.804 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-11-20T05:23:53.806 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:53.825 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:53 +0000 (0:00:00.048) 0:04:50.578 ***** 2021-11-20T05:23:53.852 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-11-20T05:23:53.854 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:53.874 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:53 +0000 (0:00:00.048) 0:04:50.626 ***** 2021-11-20T05:23:53.901 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-11-20T05:23:53.903 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:53.923 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:53 +0000 (0:00:00.048) 0:04:50.675 ***** 2021-11-20T05:23:53.952 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-11-20T05:23:53.954 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:53.974 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:53 +0000 (0:00:00.050) 0:04:50.726 ***** 2021-11-20T05:23:54.011 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-11-20T05:23:54.013 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.036 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.061) 0:04:50.788 ***** 2021-11-20T05:23:54.063 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-11-20T05:23:54.064 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.085 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.049) 0:04:50.837 ***** 2021-11-20T05:23:54.112 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-11-20T05:23:54.133 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.047) 0:04:50.885 ***** 2021-11-20T05:23:54.161 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-11-20T05:23:54.163 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.187 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.054) 0:04:50.940 ***** 2021-11-20T05:23:54.215 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-11-20T05:23:54.217 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.240 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.051) 0:04:50.992 ***** 2021-11-20T05:23:54.278 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* skipping: [smithi187.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.280 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.304 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.064) 0:04:51.056 ***** 2021-11-20T05:23:54.332 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-11-20T05:23:54.334 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.354 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.050) 0:04:51.107 ***** 2021-11-20T05:23:54.383 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-11-20T05:23:54.405 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.050) 0:04:51.158 ***** 2021-11-20T05:23:54.433 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.455 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.050) 0:04:51.208 ***** 2021-11-20T05:23:54.483 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.505 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.049) 0:04:51.257 ***** 2021-11-20T05:23:54.532 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-11-20T05:23:54.534 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.554 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.049) 0:04:51.307 ***** 2021-11-20T05:23:54.582 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-20T05:23:54.584 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.604 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.049) 0:04:51.356 ***** 2021-11-20T05:23:54.632 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-11-20T05:23:54.634 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.654 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.049) 0:04:51.406 ***** 2021-11-20T05:23:54.681 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T05:23:54.683 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.703 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.049) 0:04:51.456 ***** 2021-11-20T05:23:54.731 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-11-20T05:23:54.733 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.753 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.049) 0:04:51.505 ***** 2021-11-20T05:23:54.781 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-11-20T05:23:54.783 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.806 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.052) 0:04:51.558 ***** 2021-11-20T05:23:54.834 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-11-20T05:23:54.836 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.858 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.051) 0:04:51.610 ***** 2021-11-20T05:23:54.885 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-11-20T05:23:54.887 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.908 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.049) 0:04:51.660 ***** 2021-11-20T05:23:54.935 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-11-20T05:23:54.937 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:54.957 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:54 +0000 (0:00:00.049) 0:04:51.710 ***** 2021-11-20T05:23:54.984 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-11-20T05:23:54.986 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:55.008 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:55 +0000 (0:00:00.050) 0:04:51.760 ***** 2021-11-20T05:23:55.035 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-11-20T05:23:55.037 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:55.057 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:55 +0000 (0:00:00.049) 0:04:51.809 ***** 2021-11-20T05:23:55.084 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-11-20T05:23:55.086 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:55.107 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:55 +0000 (0:00:00.050) 0:04:51.860 ***** 2021-11-20T05:23:55.135 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-11-20T05:23:55.136 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:55.156 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:55 +0000 (0:00:00.049) 0:04:51.909 ***** 2021-11-20T05:23:55.183 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-11-20T05:23:55.185 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:55.205 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:55 +0000 (0:00:00.048) 0:04:51.957 ***** 2021-11-20T05:23:55.243 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-11-20T05:23:55.246 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:55.265 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:55 +0000 (0:00:00.060) 0:04:52.018 ***** 2021-11-20T05:23:55.292 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-11-20T05:23:55.294 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:55.314 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:55 +0000 (0:00:00.048) 0:04:52.066 ***** 2021-11-20T05:23:55.341 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-11-20T05:23:55.343 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:55.363 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:55 +0000 (0:00:00.049) 0:04:52.116 ***** 2021-11-20T05:23:55.390 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-11-20T05:23:55.392 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:55.412 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:55 +0000 (0:00:00.048) 0:04:52.164 ***** 2021-11-20T05:23:55.439 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-11-20T05:23:55.441 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:55.461 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:55 +0000 (0:00:00.048) 0:04:52.213 ***** 2021-11-20T05:23:55.488 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-11-20T05:23:55.490 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:55.509 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:55 +0000 (0:00:00.048) 0:04:52.262 ***** 2021-11-20T05:23:55.537 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-11-20T05:23:55.539 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:55.560 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:55 +0000 (0:00:00.049) 0:04:52.312 ***** 2021-11-20T05:23:55.588 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:55.611 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:55 +0000 (0:00:00.051) 0:04:52.364 ***** 2021-11-20T05:23:55.639 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-11-20T05:23:55.641 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:55.662 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:55 +0000 (0:00:00.050) 0:04:52.414 ***** 2021-11-20T05:23:55.690 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T05:23:55.692 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:55.711 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:55 +0000 (0:00:00.049) 0:04:52.464 ***** 2021-11-20T05:23:55.739 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-20T05:23:55.741 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:55.761 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:55 +0000 (0:00:00.049) 0:04:52.513 ***** 2021-11-20T05:23:55.788 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-11-20T05:23:55.790 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:55.811 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:55 +0000 (0:00:00.049) 0:04:52.563 ***** 2021-11-20T05:23:56.025 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T05:23:56.027 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-11-20T05:23:56.052 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:56 +0000 (0:00:00.241) 0:04:52.804 ***** 2021-11-20T05:23:56.257 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:23:56.279 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:56 +0000 (0:00:00.227) 0:04:53.031 ***** 2021-11-20T05:23:56.473 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T05:23:56.474 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-11-20T05:23:56.496 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:56 +0000 (0:00:00.217) 0:04:53.248 ***** 2021-11-20T05:23:56.540 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-20T05:23:56.541 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:56.567 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:56 +0000 (0:00:00.071) 0:04:53.319 ***** 2021-11-20T05:23:56.595 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-11-20T05:23:56.596 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:56.617 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:56 +0000 (0:00:00.049) 0:04:53.369 ***** 2021-11-20T05:23:56.645 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-11-20T05:23:56.647 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:56.669 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:56 +0000 (0:00:00.052) 0:04:53.422 ***** 2021-11-20T05:23:56.699 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T05:23:56.701 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:56.722 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:56 +0000 (0:00:00.052) 0:04:53.474 ***** 2021-11-20T05:23:56.750 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T05:23:56.752 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:56.772 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:56 +0000 (0:00:00.050) 0:04:53.525 ***** 2021-11-20T05:23:56.800 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T05:23:56.802 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:56.822 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:56 +0000 (0:00:00.049) 0:04:53.574 ***** 2021-11-20T05:23:56.849 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T05:23:56.851 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:56.872 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:56 +0000 (0:00:00.050) 0:04:53.625 ***** 2021-11-20T05:23:56.900 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T05:23:56.902 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:56.923 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:56 +0000 (0:00:00.050) 0:04:53.675 ***** 2021-11-20T05:23:56.950 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T05:23:56.952 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:56.974 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:56 +0000 (0:00:00.050) 0:04:53.726 ***** 2021-11-20T05:23:57.001 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.023 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:57 +0000 (0:00:00.049) 0:04:53.776 ***** 2021-11-20T05:23:57.051 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-11-20T05:23:57.053 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.075 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:57 +0000 (0:00:00.051) 0:04:53.828 ***** 2021-11-20T05:23:57.102 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-20T05:23:57.104 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.124 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:57 +0000 (0:00:00.048) 0:04:53.876 ***** 2021-11-20T05:23:57.150 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-11-20T05:23:57.152 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.172 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:57 +0000 (0:00:00.047) 0:04:53.924 ***** 2021-11-20T05:23:57.199 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.222 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:57 +0000 (0:00:00.050) 0:04:53.975 ***** 2021-11-20T05:23:57.250 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.283 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:57 +0000 (0:00:00.060) 0:04:54.035 ***** 2021-11-20T05:23:57.313 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-11-20T05:23:57.315 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.336 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:57 +0000 (0:00:00.052) 0:04:54.088 ***** 2021-11-20T05:23:57.363 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-11-20T05:23:57.365 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.386 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:57 +0000 (0:00:00.050) 0:04:54.138 ***** 2021-11-20T05:23:57.413 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.435 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:57 +0000 (0:00:00.048) 0:04:54.187 ***** 2021-11-20T05:23:57.462 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-11-20T05:23:57.463 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.484 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:57 +0000 (0:00:00.048) 0:04:54.236 ***** 2021-11-20T05:23:57.511 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-11-20T05:23:57.512 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.532 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:57 +0000 (0:00:00.048) 0:04:54.285 ***** 2021-11-20T05:23:57.560 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-11-20T05:23:57.561 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.582 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:57 +0000 (0:00:00.049) 0:04:54.334 ***** 2021-11-20T05:23:57.610 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-11-20T05:23:57.612 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.632 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:57 +0000 (0:00:00.049) 0:04:54.384 ***** 2021-11-20T05:23:57.670 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:23:57.672 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-11-20T05:23:57.692 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:57 +0000 (0:00:00.060) 0:04:54.445 ***** 2021-11-20T05:23:57.731 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-11-20T05:23:57.732 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.756 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi187.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.766 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.777 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.779 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.788 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-11-20T05:23:57.814 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:57 +0000 (0:00:00.121) 0:04:54.566 ***** 2021-11-20T05:23:58.008 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-11-20T05:23:58.010 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003030", "end": "2021-11-20 05:23:57.983803", "rc": 0, "start": "2021-11-20 05:23:57.980773", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock"]} 2021-11-20T05:23:58.033 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:58 +0000 (0:00:00.219) 0:04:54.786 ***** 2021-11-20T05:23:58.064 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:58.089 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:58 +0000 (0:00:00.055) 0:04:54.841 ***** 2021-11-20T05:23:58.116 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-11-20T05:23:58.118 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:58.139 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:58 +0000 (0:00:00.049) 0:04:54.891 ***** 2021-11-20T05:23:58.166 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-11-20T05:23:58.168 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:58.188 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:58 +0000 (0:00:00.049) 0:04:54.941 ***** 2021-11-20T05:23:58.215 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-11-20T05:23:58.218 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:23:58.237 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:23:58 +0000 (0:00:00.048) 0:04:54.990 ***** 2021-11-20T05:24:00.237 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-11-20T05:24:00.239 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:24:00.259 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:00 +0000 (0:00:02.022) 0:04:57.012 ***** 2021-11-20T05:24:00.298 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-11-20T05:24:00.300 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140188672, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085007, 'block_used': 13010101, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085007, "block_size": 4096, "block_total": 240095108, "block_used": 13010101, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140188672, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}} 2021-11-20T05:24:00.321 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:00 +0000 (0:00:00.061) 0:04:57.074 ***** 2021-11-20T05:24:00.517 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005236", "end": "2021-11-20 05:24:00.492740", "rc": 0, "start": "2021-11-20 05:24:00.487504", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:00.539 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:00 +0000 (0:00:00.218) 0:04:57.292 ***** 2021-11-20T05:24:00.578 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-11-20T05:24:00.580 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140188672, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085007, 'block_used': 13010101, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085007, "block_size": 4096, "block_total": 240095108, "block_used": 13010101, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140188672, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:00.602 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:00 +0000 (0:00:00.062) 0:04:57.355 ***** 2021-11-20T05:24:00.630 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-11-20T05:24:00.653 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:00 +0000 (0:00:00.050) 0:04:57.405 ***** 2021-11-20T05:24:00.923 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-20T05:24:00.926 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: systemd-pam-239-51.el8.x86_64", "Installed: systemd-239-51.el8.x86_64", "Installed: systemd-libs-239-51.el8.x86_64", "Installed: systemd-udev-239-51.el8.x86_64", "Removed: systemd-239-45.el8.x86_64", "Removed: systemd-libs-239-45.el8.x86_64", "Removed: systemd-pam-239-45.el8.x86_64", "Removed: systemd-udev-239-45.el8.x86_64"]} 2021-11-20T05:24:00.948 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:00 +0000 (0:00:00.295) 0:04:57.700 ***** 2021-11-20T05:24:01.270 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-11-20T05:24:01.271 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.socket polkit.service dbus.service basic.target system.slice sysinit.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target iptables.service ebtables.service ip6tables.service ipset.service nftables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate" 2021-11-20T05:24:01.272 INFO:teuthology.task.ansible.out:: "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": 2021-11-20T05:24:01.273 INFO:teuthology.task.ansible.out:"0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127267", "LimitNPROCSoft": "127267", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127267", "LimitSIGPENDINGSoft": "127267", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket system.slice sysinit.target", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "nul 2021-11-20T05:24:01.275 INFO:teuthology.task.ansible.out:l", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203627", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-11-20T05:24:01.295 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:01 +0000 (0:00:00.347) 0:04:58.048 ***** 2021-11-20T05:24:01.596 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-11-20T05:24:01.598 INFO:teuthology.task.ansible.out:fatal: [smithi058.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-11-20T05:24:01.619 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:01 +0000 (0:00:00.323) 0:04:58.372 ***** 2021-11-20T05:24:01.921 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-11-20T05:24:01.948 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:01 +0000 (0:00:00.328) 0:04:58.700 ***** 2021-11-20T05:24:01.978 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-11-20T05:24:01.980 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.002 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:01 +0000 (0:00:00.054) 0:04:58.754 ***** 2021-11-20T05:24:02.030 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-11-20T05:24:02.032 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.054 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.051) 0:04:58.805 ***** 2021-11-20T05:24:02.082 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.105 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.051) 0:04:58.857 ***** 2021-11-20T05:24:02.133 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-11-20T05:24:02.137 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.155 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.050) 0:04:58.908 ***** 2021-11-20T05:24:02.182 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-11-20T05:24:02.184 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.204 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.049) 0:04:58.957 ***** 2021-11-20T05:24:02.232 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-11-20T05:24:02.234 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.256 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.051) 0:04:59.008 ***** 2021-11-20T05:24:02.284 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-11-20T05:24:02.310 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.053) 0:04:59.062 ***** 2021-11-20T05:24:02.340 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-11-20T05:24:02.342 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.363 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.053) 0:04:59.116 ***** 2021-11-20T05:24:02.391 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-11-20T05:24:02.393 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.413 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.050) 0:04:59.166 ***** 2021-11-20T05:24:02.453 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* skipping: [smithi058.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.465 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-20T05:24:02.466 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djtQK1bhqyXJgXZnE5ciNBhJ9SKkHTqmPI7'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.589184", "end": "2021-11-20 05:24:02.426555", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djtQK1bhqyXJgXZnE5ciNBhJ9SKkHTqmPI7"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:00.837371", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} TASK [testnode : Install apt keys] ********************************************* skipping: [smithi058.front.sepia.ceph. 2021-11-20T05:24:02.468 INFO:teuthology.task.ansible.out:com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.502 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.088) 0:04:59.254 ***** 2021-11-20T05:24:02.530 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-11-20T05:24:02.532 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.551 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.049) 0:04:59.304 ***** 2021-11-20T05:24:02.578 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-11-20T05:24:02.600 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.048) 0:04:59.352 ***** 2021-11-20T05:24:02.627 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-11-20T05:24:02.629 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.650 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.050) 0:04:59.403 ***** 2021-11-20T05:24:02.677 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.699 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.048) 0:04:59.451 ***** 2021-11-20T05:24:02.726 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-11-20T05:24:02.728 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.748 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.049) 0:04:59.501 ***** 2021-11-20T05:24:02.777 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.798 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.050) 0:04:59.551 ***** 2021-11-20T05:24:02.826 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-11-20T05:24:02.827 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.847 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.048) 0:04:59.600 ***** 2021-11-20T05:24:02.874 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T05:24:02.876 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.896 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.048) 0:04:59.649 ***** 2021-11-20T05:24:02.923 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-11-20T05:24:02.925 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.947 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.048) 0:04:59.697 ***** 2021-11-20T05:24:02.973 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-11-20T05:24:02.975 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:02.995 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:02 +0000 (0:00:00.049) 0:04:59.747 ***** 2021-11-20T05:24:03.025 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-11-20T05:24:03.028 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.047 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.052) 0:04:59.800 ***** 2021-11-20T05:24:03.075 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-11-20T05:24:03.077 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.099 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.051) 0:04:59.851 ***** 2021-11-20T05:24:03.126 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-11-20T05:24:03.128 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.149 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.050) 0:04:59.901 ***** 2021-11-20T05:24:03.176 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.201 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.052) 0:04:59.953 ***** 2021-11-20T05:24:03.229 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.251 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.050) 0:05:00.004 ***** 2021-11-20T05:24:03.279 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.302 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.050) 0:05:00.054 ***** 2021-11-20T05:24:03.330 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-11-20T05:24:03.332 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.352 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.050) 0:05:00.105 ***** 2021-11-20T05:24:03.381 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-11-20T05:24:03.383 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.403 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.050) 0:05:00.156 ***** 2021-11-20T05:24:03.430 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-11-20T05:24:03.432 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.453 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.050) 0:05:00.206 ***** 2021-11-20T05:24:03.481 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-11-20T05:24:03.482 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.503 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.049) 0:05:00.256 ***** 2021-11-20T05:24:03.530 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-11-20T05:24:03.532 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.553 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.050) 0:05:00.306 ***** 2021-11-20T05:24:03.581 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-11-20T05:24:03.583 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.603 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.049) 0:05:00.356 ***** 2021-11-20T05:24:03.631 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-11-20T05:24:03.633 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.654 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.051) 0:05:00.407 ***** 2021-11-20T05:24:03.681 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-11-20T05:24:03.683 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.705 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.050) 0:05:00.457 ***** 2021-11-20T05:24:03.712 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-20T05:24:03.713 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-mJvGZE-ehMz-Nw96-ZWBF-dDEn-jPwr-niRiD7', 'nvme-INTEL_SSDPEDMD400G4_PHFT620500MW400BGN', 'nvme-nvme.8086-504846543632303530304d5734303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.044396", "end": "2021-11-20 05:24:03.660033", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-mJvGZE-ehMz-Nw96-ZWBF-dDEn-jPwr-niRiD7", "nvme-INTEL_SSDPEDMD400G4_PHFT620500MW400BGN", "nvme-nvme.8086-504846543632303530304d5734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:02.615637", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-11-20T05:24:03.715 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:24:03.735 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-11-20T05:24:03.737 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.758 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.052) 0:05:00.510 ***** 2021-11-20T05:24:03.786 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:24:03.787 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.809 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.049) 0:05:00.560 ***** 2021-11-20T05:24:03.835 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-11-20T05:24:03.837 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.858 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.049) 0:05:00.610 ***** 2021-11-20T05:24:03.886 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.911 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.053) 0:05:00.663 ***** 2021-11-20T05:24:03.939 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:03.961 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:03 +0000 (0:00:00.050) 0:05:00.714 ***** 2021-11-20T05:24:03.989 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:04.012 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:04 +0000 (0:00:00.051) 0:05:00.765 ***** 2021-11-20T05:24:04.239 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T05:24:04.241 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-11-20T05:24:04.262 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:04 +0000 (0:00:00.249) 0:05:01.014 ***** 2021-11-20T05:24:04.532 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T05:24:04.534 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:24:04.553 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:04 +0000 (0:00:00.291) 0:05:01.306 ***** 2021-11-20T05:24:04.759 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T05:24:04.761 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-11-20T05:24:04.782 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:04 +0000 (0:00:00.228) 0:05:01.535 ***** 2021-11-20T05:24:04.809 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-20T05:24:04.811 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:04.831 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:04 +0000 (0:00:00.049) 0:05:01.584 ***** 2021-11-20T05:24:04.859 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-11-20T05:24:04.861 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:04.883 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:04 +0000 (0:00:00.051) 0:05:01.635 ***** 2021-11-20T05:24:04.912 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:04.933 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:04 +0000 (0:00:00.050) 0:05:01.686 ***** 2021-11-20T05:24:04.951 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-20T05:24:04.953 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djt2idIXPbbk7mp46dFVw1BO2N7zLcgm0zG'], 'uuids': ['d2ae65c8-afdd-4ab1-8aca-2c2381a6079f'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.077138", "end": "2021-11-20 05:24:04.927834", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djt2idIXPbbk7mp46dFVw1BO2N7zLcgm0zG"], "labels": [], "masters": [], "uuids": ["d2ae65c8-afdd-4ab1-8aca-2c2381a6079f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:03.850696", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:24:04.964 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T05:24:04.965 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:04.986 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:04 +0000 (0:00:00.052) 0:05:01.739 ***** 2021-11-20T05:24:05.014 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T05:24:05.016 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:05.037 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:05 +0000 (0:00:00.051) 0:05:01.790 ***** 2021-11-20T05:24:05.066 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T05:24:05.068 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:05.089 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:05 +0000 (0:00:00.052) 0:05:01.842 ***** 2021-11-20T05:24:05.118 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:05.144 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:05 +0000 (0:00:00.054) 0:05:01.896 ***** 2021-11-20T05:24:05.172 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T05:24:05.174 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:05.365 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:05 +0000 (0:00:00.220) 0:05:02.117 ***** 2021-11-20T05:24:05.393 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T05:24:05.395 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:05.415 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:05 +0000 (0:00:00.050) 0:05:02.167 ***** 2021-11-20T05:24:05.443 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-11-20T05:24:05.444 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:05.466 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:05 +0000 (0:00:00.050) 0:05:02.218 ***** 2021-11-20T05:24:05.493 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-11-20T05:24:05.495 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:05.515 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:05 +0000 (0:00:00.049) 0:05:02.268 ***** 2021-11-20T05:24:05.542 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-20T05:24:05.544 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:05.564 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:05 +0000 (0:00:00.048) 0:05:02.317 ***** 2021-11-20T05:24:05.593 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-11-20T05:24:05.595 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:05.615 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:05 +0000 (0:00:00.051) 0:05:02.368 ***** 2021-11-20T05:24:05.653 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:05.675 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:05 +0000 (0:00:00.059) 0:05:02.428 ***** 2021-11-20T05:24:05.703 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:05.734 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:05 +0000 (0:00:00.059) 0:05:02.487 ***** 2021-11-20T05:24:05.765 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-11-20T05:24:05.766 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:05.787 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:05 +0000 (0:00:00.052) 0:05:02.540 ***** 2021-11-20T05:24:05.814 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-11-20T05:24:05.816 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:05.837 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:05 +0000 (0:00:00.049) 0:05:02.589 ***** 2021-11-20T05:24:05.866 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:05.897 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:05 +0000 (0:00:00.059) 0:05:02.649 ***** 2021-11-20T05:24:05.927 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-11-20T05:24:05.929 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:05.953 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:05 +0000 (0:00:00.055) 0:05:02.705 ***** 2021-11-20T05:24:05.983 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:06.005 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:06 +0000 (0:00:00.052) 0:05:02.758 ***** 2021-11-20T05:24:06.033 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-11-20T05:24:06.035 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:06.055 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:06 +0000 (0:00:00.049) 0:05:02.808 ***** 2021-11-20T05:24:06.082 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-11-20T05:24:06.084 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:06.107 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:06 +0000 (0:00:00.052) 0:05:02.860 ***** 2021-11-20T05:24:06.145 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:24:06.147 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-11-20T05:24:06.167 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:06 +0000 (0:00:00.059) 0:05:02.919 ***** 2021-11-20T05:24:06.205 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-20T05:24:06.207 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djtkRNnRRw7ypSpeupLDREq4UtqurY0un3x'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.052898", "end": "2021-11-20 05:24:06.170851", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djtkRNnRRw7ypSpeupLDREq4UtqurY0un3x"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:05.117953", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:24:06.227 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-11-20T05:24:06.228 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-11-20T05:24:06.249 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-11-20T05:24:06.250 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi058.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-11-20T05:24:06.261 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-11-20T05:24:06.262 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-11-20T05:24:06.272 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-11-20T05:24:06.297 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:06 +0000 (0:00:00.129) 0:05:03.049 ***** 2021-11-20T05:24:06.493 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-11-20T05:24:06.495 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004142", "end": "2021-11-20 05:24:06.470164", "rc": 0, "start": "2021-11-20 05:24:06.466022", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock"]} 2021-11-20T05:24:06.518 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:06 +0000 (0:00:00.220) 0:05:03.270 ***** 2021-11-20T05:24:06.545 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-11-20T05:24:06.547 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:06.570 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:06 +0000 (0:00:00.052) 0:05:03.323 ***** 2021-11-20T05:24:06.597 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-11-20T05:24:06.599 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:06.620 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:06 +0000 (0:00:00.049) 0:05:03.372 ***** 2021-11-20T05:24:06.647 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-11-20T05:24:06.649 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:06.669 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:06 +0000 (0:00:00.049) 0:05:03.422 ***** 2021-11-20T05:24:06.697 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:06.718 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:06 +0000 (0:00:00.049) 0:05:03.471 ***** 2021-11-20T05:24:07.445 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-20T05:24:07.447 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djtuEIQ5W4rdNUB7CHFnFozYrV1UsZN6RZ7'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.057308", "end": "2021-11-20 05:24:07.413025", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djtuEIQ5W4rdNUB7CHFnFozYrV1UsZN6RZ7"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:06.355717", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:24:07.448 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM 2021-11-20T05:24:07.449 INFO:teuthology.task.ansible.out:0033-9ZM173_Z1W5XFQE', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFQE', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFQE', 'scsi-35000c5009292232d', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFQE', 'wwn-0x5000c5009292232d'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XFQE', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009292232d', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFQE-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFQE-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFQE-part1', 'scsi-35000c5009292232d-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFQE-part1', 'wwn-0x5000c5009292232d-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFQE", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFQE", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFQE", "scsi-35000c5009292232d", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFQE", "wwn-0x5000c5009292232d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFQE-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFQE-part1", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFQE-part1", "scsi-35000c5009292232d-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFQE-part1", "wwn-0x5000c5009292232d-part 2021-11-20T05:24:07.451 INFO:teuthology.task.ansible.out:1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFQE", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009292232d"}}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:08.749 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djtmPj0ASSz5D6KXxTjF76wTBXd2OiMmekF'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.112934", "end": "2021-11-20 05:24:08.718888", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djtmPj0ASSz5D6KXxTjF76wTBXd2OiMmekF"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:07.605954", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:24:08.782 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:08 +0000 (0:00:02.063) 0:05:05.535 ***** 2021-11-20T05:24:08.821 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-11-20T05:24:08.822 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:24:08.844 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:08 +0000 (0:00:00.061) 0:05:05.596 ***** 2021-11-20T05:24:08.893 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-11-20T05:24:08.895 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140073984, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227084979, 'block_used': 13010129, 'inode_total': 61054976, 'inode_available': 60984950, 'inode_used': 70026, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227084979, "block_size": 4096, "block_total": 240095108, "block_used": 13010129, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984950, "inode_total": 61054976, "inode_used": 70026, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140073984, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}} 2021-11-20T05:24:08.917 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:08 +0000 (0:00:00.073) 0:05:05.670 ***** 2021-11-20T05:24:09.123 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-11-20T05:24:09.125 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005455", "end": "2021-11-20 05:24:09.103865", "rc": 0, "start": "2021-11-20 05:24:09.098410", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:09.146 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:09 +0000 (0:00:00.229) 0:05:05.899 ***** 2021-11-20T05:24:09.165 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-11-20T05:24:09.167 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djtQK1bhqyXJgXZnE5ciNBhJ9SKkHTqmPI7'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.153288", "end": "2021-11-20 05:24:09.131740", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djtQK1bhqyXJgXZnE5ciNBhJ9SKkHTqmPI7"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:08.978452", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150494 s, 766 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150494 s, 766 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:09.190 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi058.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140073984, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227084979, 'block_used': 13010129, 'inode_total': 61054976, 'inode_available': 60984950, 'inode_used': 70026, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227084979, "block_size": 4096, "block_total": 240095108, "block_used": 13010129, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984950, "inode_total": 61054976, "inode_used": 70026, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140073984, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:09.211 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:09 +0000 (0:00:00.064) 0:05:05.964 ***** 2021-11-20T05:24:09.238 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-11-20T05:24:09.260 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:09 +0000 (0:00:00.048) 0:05:06.012 ***** 2021-11-20T05:24:09.371 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-11-20T05:24:09.372 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-mJvGZE-ehMz-Nw96-ZWBF-dDEn-jPwr-niRiD7', 'nvme-INTEL_SSDPEDMD400G4_PHFT620500MW400BGN', 'nvme-nvme.8086-504846543632303530304d5734303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.038837", "end": "2021-11-20 05:24:09.347565", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-mJvGZE-ehMz-Nw96-ZWBF-dDEn-jPwr-niRiD7", "nvme-INTEL_SSDPEDMD400G4_PHFT620500MW400BGN", "nvme-nvme.8086-504846543632303530304d5734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:09.308728", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0360714 s, 3.2 GB/s", "std 2021-11-20T05:24:09.374 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0360714 s, 3.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:09.707 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djt2idIXPbbk7mp46dFVw1BO2N7zLcgm0zG'], 'uuids': ['d2ae65c8-afdd-4ab1-8aca-2c2381a6079f'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.141562", "end": "2021-11-20 05:24:09.675934", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djt2idIXPbbk7mp46dFVw1BO2N7zLcgm0zG"], "labels": [], "masters": [], "uuids": ["d2ae65c8-afdd-4ab1-8aca-2c2381a6079f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:09.534372", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.13893 s, 830 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.13893 s, 830 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:10.021 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djtkRNnRRw7ypSpeupLDREq4UtqurY0un3x'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.141087", "end": "2021-11-20 05:24:09.996166", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djtkRNnRRw7ypSpeupLDREq4UtqurY0un3x"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:09.855079", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138423 s, 833 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138423 s, 833 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:10.342 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djtuEIQ5W4rdNUB7CHFnFozYrV1UsZN6RZ7'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.143103", "end": "2021-11-20 05:24:10.317332", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djtuEIQ5W4rdNUB7CHFnFozYrV1UsZN6RZ7"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:10.174229", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140481 s, 821 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140481 s, 821 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:10.344 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFQE', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFQE', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFQE', 'scsi-35000c5009292232d', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFQE', 'wwn-0x5000c5009292232d'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XFQE', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009292232d', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFQE-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFQE-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFQE-part1', 'scsi-35000c5009292232d-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFQE-part1', 'wwn-0x5000c5009292232d-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFQE", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFQE", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFQE", "scsi-35000c5009292232d", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFQE", "wwn-0x5000c5009292232d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFQE-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFQE-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-11-20T05:24:10.346 INFO:teuthology.task.ansible.out:Z1W5XFQE-part1", "scsi-35000c5009292232d-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFQE-part1", "wwn-0x5000c5009292232d-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFQE", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009292232d"}}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:10.666 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djtmPj0ASSz5D6KXxTjF76wTBXd2OiMmekF'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.142629", "end": "2021-11-20 05:24:10.637607", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-GJr5guvfrcalBfztvA0QMVB7KCL66djtmPj0ASSz5D6KXxTjF76wTBXd2OiMmekF"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:10.494978", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139995 s, 824 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139995 s, 824 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:10.694 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:10 +0000 (0:00:01.434) 0:05:07.447 ***** 2021-11-20T05:24:10.901 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-20T05:24:10.902 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5SxErJxeojdQmM2KRmoHgpr1xy620EEXRf'], 'uuids': ['699f501c-02ae-422d-a2dc-048d975d2e4d'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.412110", "end": "2021-11-20 05:24:10.867003", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5SxErJxeojdQmM2KRmoHgpr1xy620EEXRf"], "labels": [], "masters": [], "uuids": ["699f501c-02ae-422d-a2dc-048d975d2e4d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:09.454893", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:24:10.934 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-11-20T05:24:10.936 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.048676", "end": "2021-11-20 05:24:10.907091", "rc": 0, "start": "2021-11-20 05:24:10.858415", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:10.959 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:10 +0000 (0:00:00.264) 0:05:07.711 ***** 2021-11-20T05:24:11.208 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-11-20T05:24:11.210 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.059571", "end": "2021-11-20 05:24:11.181970", "rc": 0, "start": "2021-11-20 05:24:11.122399", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:11.237 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:11 +0000 (0:00:00.277) 0:05:07.989 ***** 2021-11-20T05:24:11.274 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-11-20T05:24:11.302 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:11 +0000 (0:00:00.065) 0:05:08.054 ***** 2021-11-20T05:24:11.329 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-11-20T05:24:11.331 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:11.358 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:11 +0000 (0:00:00.055) 0:05:08.110 ***** 2021-11-20T05:24:11.385 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-11-20T05:24:11.387 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:11.409 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:11 +0000 (0:00:00.051) 0:05:08.161 ***** 2021-11-20T05:24:11.437 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-11-20T05:24:11.439 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:11.464 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:11 +0000 (0:00:00.055) 0:05:08.217 ***** 2021-11-20T05:24:11.492 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-11-20T05:24:11.494 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:11.516 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:11 +0000 (0:00:00.051) 0:05:08.269 ***** 2021-11-20T05:24:11.701 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-11-20T05:24:11.703 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:24:11.724 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:11 +0000 (0:00:00.207) 0:05:08.477 ***** 2021-11-20T05:24:11.762 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-11-20T05:24:11.764 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140188672, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085007, 'block_used': 13010101, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085007, "block_size": 4096, "block_total": 240095108, "block_used": 13010101, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140188672, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:11.785 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:11 +0000 (0:00:00.060) 0:05:08.538 ***** 2021-11-20T05:24:11.813 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-11-20T05:24:11.814 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:11.835 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:11 +0000 (0:00:00.049) 0:05:08.587 ***** 2021-11-20T05:24:11.873 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:24:11.874 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:11.895 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:11 +0000 (0:00:00.060) 0:05:08.648 ***** 2021-11-20T05:24:11.933 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:24:11.934 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:11.955 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:11 +0000 (0:00:00.060) 0:05:08.708 ***** 2021-11-20T05:24:11.993 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:12.016 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:12 +0000 (0:00:00.060) 0:05:08.769 ***** 2021-11-20T05:24:12.054 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-11-20T05:24:12.056 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:12.084 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:12 +0000 (0:00:00.067) 0:05:08.836 ***** 2021-11-20T05:24:12.112 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-11-20T05:24:12.114 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:12.138 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:12 +0000 (0:00:00.054) 0:05:08.891 ***** 2021-11-20T05:24:12.156 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-20T05:24:12.158 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-Pv0h1p-i98C-BX77-1sQn-Hw7a-iNJY-e10Rff', 'nvme-INTEL_SSDPED1D480GA_PHMB751300GN480DGN', 'nvme-nvme.8086-50484d42373531333030474e34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.046815", "end": "2021-11-20 05:24:12.111634", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-Pv0h1p-i98C-BX77-1sQn-Hw7a-iNJY-e10Rff", "nvme-INTEL_SSDPED1D480GA_PHMB751300GN480DGN", "nvme-nvme.8086-50484d42373531333030474e34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:11.064819", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2021-11-20T05:24:12.160 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:24:12.628 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-11-20T05:24:12.630 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-11-20T05:24:12.650 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:12 +0000 (0:00:00.511) 0:05:09.403 ***** 2021-11-20T05:24:13.115 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-11-20T05:24:13.117 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:24:13.417 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-20T05:24:13.418 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5S1LpA1A3GutdACgOxHtUwEjciENFcedHG'], 'uuids': ['d4b048d0-530a-4dd0-b79b-ed088bdb40c2'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.078271", "end": "2021-11-20 05:24:13.390197", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5S1LpA1A3GutdACgOxHtUwEjciENFcedHG"], "labels": [], "masters": [], "uuids": ["d4b048d0-530a-4dd0-b79b-ed088bdb40c2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:12.311926", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:24:13.449 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-11-20T05:24:13.451 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:24:13.804 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:24:14.136 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:24:14.470 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:24:14.495 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:14 +0000 (0:00:01.844) 0:05:11.248 ***** 2021-11-20T05:24:14.673 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-20T05:24:14.674 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5SUP57dDziZP2yYRyF3Xf28gUp3N8NxPF5'], 'uuids': ['20c42b36-20fe-4150-97ea-2830ed626f09'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.051718", "end": "2021-11-20 05:24:14.640174", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5SUP57dDziZP2yYRyF3Xf28gUp3N8NxPF5"], "labels": [], "masters": [], "uuids": ["20c42b36-20fe-4150-97ea-2830ed626f09"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:13.588456", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:24:14.696 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-11-20T05:24:14.698 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-11-20T05:24:14.719 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:14 +0000 (0:00:00.223) 0:05:11.471 ***** 2021-11-20T05:24:14.935 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-11-20T05:24:14.937 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T05:24:15.104 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T05:24:15.292 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T05:24:15.462 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T05:24:15.473 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:15.497 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:15 +0000 (0:00:00.778) 0:05:12.250 ***** 2021-11-20T05:24:15.524 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-11-20T05:24:15.526 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:15.546 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:15 +0000 (0:00:00.048) 0:05:12.299 ***** 2021-11-20T05:24:15.573 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-11-20T05:24:15.575 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:15.595 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:15 +0000 (0:00:00.049) 0:05:12.348 ***** 2021-11-20T05:24:15.789 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-11-20T05:24:15.791 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:24:15.812 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:15 +0000 (0:00:00.216) 0:05:12.564 ***** 2021-11-20T05:24:15.861 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-11-20T05:24:15.863 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-11-20T05:24:15.885 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:15 +0000 (0:00:00.073) 0:05:12.638 ***** 2021-11-20T05:24:15.924 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5STPq8Vl1uoBJAq12DCp5bgPWjFuqYGk1e'], 'uuids': ['069d4c61-0dc3-40af-9f4f-82e6ddeb42c7'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.060639", "end": "2021-11-20 05:24:15.900522", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5STPq8Vl1uoBJAq12DCp5bgPWjFuqYGk1e"], "labels": [], "masters": [], "uuids": ["069d4c61-0dc3-40af-9f4f-82e6ddeb42c7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:14.839883", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk u 2021-11-20T05:24:15.927 INFO:teuthology.task.ansible.out:sing fdisk or", "other utilities."]} 2021-11-20T05:24:15.936 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQFR', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQFR', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQFR', 'scsi-35000c500871a7ce9', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HQFR', 'wwn-0x5000c500871a7ce9'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4HQFR', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a7ce9', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQFR-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQFR-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQFR-part1', 'scsi-35000c500871a7ce9-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HQFR-part1', 'wwn-0x5000c500871a7ce9-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQFR", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQFR", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQFR", "scsi-35000c500871a7ce9", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HQFR", "wwn-0x5000c500871a7ce9"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQFR-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQFR-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-11-20T05:24:15.938 INFO:teuthology.task.ansible.out:Z1W4HQFR-part1", "scsi-35000c500871a7ce9-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HQFR-part1", "wwn-0x5000c500871a7ce9-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HQFR", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a7ce9"}}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:16.242 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi187.front.sepia.ceph.com] => {"changed": true} 2021-11-20T05:24:16.266 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:16 +0000 (0:00:00.380) 0:05:13.019 ***** 2021-11-20T05:24:16.680 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-11-20T05:24:16.682 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-11-20T05:24:16.703 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:16 +0000 (0:00:00.436) 0:05:13.456 ***** 2021-11-20T05:24:16.730 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-11-20T05:24:16.732 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:16.752 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:16 +0000 (0:00:00.049) 0:05:13.505 ***** 2021-11-20T05:24:16.780 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:16.802 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:16 +0000 (0:00:00.049) 0:05:13.555 ***** 2021-11-20T05:24:16.829 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-11-20T05:24:16.831 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:16.863 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:16 +0000 (0:00:00.060) 0:05:13.615 ***** 2021-11-20T05:24:16.893 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-11-20T05:24:16.895 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:16.915 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:16 +0000 (0:00:00.052) 0:05:13.668 ***** 2021-11-20T05:24:16.943 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-11-20T05:24:16.945 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:16.965 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:16 +0000 (0:00:00.049) 0:05:13.718 ***** 2021-11-20T05:24:16.992 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:17.016 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:17 +0000 (0:00:00.051) 0:05:13.769 ***** 2021-11-20T05:24:17.043 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-11-20T05:24:17.045 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:17.065 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:17 +0000 (0:00:00.048) 0:05:13.817 ***** 2021-11-20T05:24:17.092 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-11-20T05:24:17.094 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:17.114 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:17 +0000 (0:00:00.049) 0:05:13.867 ***** 2021-11-20T05:24:17.142 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-11-20T05:24:17.144 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:17.164 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:17 +0000 (0:00:00.049) 0:05:13.917 ***** 2021-11-20T05:24:17.191 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-11-20T05:24:17.193 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:17.224 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:17 +0000 (0:00:00.060) 0:05:13.977 ***** 2021-11-20T05:24:17.236 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-20T05:24:17.238 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5S4VlicT1MXjDi5AilltMPww5X2kbhLE3w'], 'uuids': ['bfea6be8-cc5a-42e5-b893-1f0800dc443c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.073106", "end": "2021-11-20 05:24:17.178596", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5S4VlicT1MXjDi5AilltMPww5X2kbhLE3w"], "labels": [], "masters": [], "uuids": ["bfea6be8-cc5a-42e5-b893-1f0800dc443c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:16.105490", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:24:17.284 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:17 +0000 (0:00:00.059) 0:05:14.037 ***** 2021-11-20T05:24:17.295 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-11-20T05:24:17.297 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:17.320 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:17 +0000 (0:00:00.035) 0:05:14.073 ***** 2021-11-20T05:24:17.348 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-11-20T05:24:17.350 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:17.372 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:17 +0000 (0:00:00.051) 0:05:14.124 ***** 2021-11-20T05:24:17.399 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-11-20T05:24:17.401 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:17.432 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:17 +0000 (0:00:00.059) 0:05:14.184 ***** 2021-11-20T05:24:17.459 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-11-20T05:24:17.461 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:17.482 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:17 +0000 (0:00:00.049) 0:05:14.234 ***** 2021-11-20T05:24:17.508 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-11-20T05:24:17.510 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:17.530 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:17 +0000 (0:00:00.048) 0:05:14.283 ***** 2021-11-20T05:24:17.622 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-11-20T05:24:17.624 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5SxErJxeojdQmM2KRmoHgpr1xy620EEXRf'], 'uuids': ['699f501c-02ae-422d-a2dc-048d975d2e4d'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.100735", "end": "2021-11-20 05:24:17.591011", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5SxErJxeojdQmM2KRmoHgpr1xy620EEXRf"], "labels": [], "masters": [], "uuids": ["699f501c-02ae-422d-a2dc-048d975d2e4d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:17.490276", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0978005 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0978005 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:17.834 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-Pv0h1p-i98C-BX77-1sQn-Hw7a-iNJY-e10Rff', 'nvme-INTEL_SSDPED1D480GA_PHMB751300GN480DGN', 'nvme-nvme.8086-50484d42373531333030474e34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039023", "end": "2021-11-20 05:24:17.812378", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-Pv0h1p-i98C-BX77-1sQn-Hw7a-iNJY-e10Rff", "nvme-INTEL_SSDPED1D480GA_PHMB751300GN480DGN", "nvme-nvme.8086-50484d42373531333030474e34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:17.773355", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0362012 s, 3.2 GB/s", "stderr_lines": [" 2021-11-20T05:24:17.836 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0362012 s, 3.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:18.012 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-11-20T05:24:18.014 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-11-20T05:24:18.034 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:18 +0000 (0:00:00.503) 0:05:14.787 ***** 2021-11-20T05:24:18.114 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-11-20T05:24:18.117 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5S1LpA1A3GutdACgOxHtUwEjciENFcedHG'], 'uuids': ['d4b048d0-530a-4dd0-b79b-ed088bdb40c2'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.098149", "end": "2021-11-20 05:24:18.091747", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5S1LpA1A3GutdACgOxHtUwEjciENFcedHG"], "labels": [], "masters": [], "uuids": ["d4b048d0-530a-4dd0-b79b-ed088bdb40c2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:17.993598", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0953513 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0953513 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:18.337 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-11-20T05:24:18.338 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-11-20 05:18:07 UTC", "ActiveEnterTimestampMonotonic": "12225565", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount systemd-journald.socket ntpd.service tmp.mount system.slice ntpdate.service sntp.service basic.target systemd-tmpfiles-setup.service sysinit.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-11-20 05:18:06 UTC", "AssertTimestampMonotonic": "11053839", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-11-20 05:18:06 UTC", "ConditionTimestampMonotonic": "11053777", "ConfigurationDirectoryMode": "0755", "Conflicts": "syst 2021-11-20T05:24:18.340 INFO:teuthology.task.ansible.out:emd-timesyncd.service ntpd.service shutdown.target", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "889", "ExecMainStartTimestamp": "Sat 2021-11-20 05:18:07 UTC", "ExecMainStartTimestampMonotonic": "12170484", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-11-20 05:18:06 UTC", "InactiveExitTimestampMonotonic": "11061922", "InvocationID": "39cfc765f3d342d8b5894718d7751555", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": 2021-11-20T05:24:18.342 INFO:teuthology.task.ansible.out:"infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "889", "MemoryAccounting": "yes", "MemoryCurrent": "1732608", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount syste 2021-11-20T05:24:18.344 INFO:teuthology.task.ansible.out:m.slice", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-11-20 05:18:07 UTC", "StateChangeTimestampMonotonic": "12225565", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-11-20 05:18:07 UTC", "WatchdogTimestampMonotonic": "12170496", "WatchdogUSec": "0"}} 2021-11-20T05:24:18.362 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:18 +0000 (0:00:00.327) 0:05:15.114 ***** 2021-11-20T05:24:18.389 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:24:18.390 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-11-20T05:24:18.414 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:18 +0000 (0:00:00.052) 0:05:15.167 ***** 2021-11-20T05:24:18.421 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-11-20T05:24:18.422 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5SUP57dDziZP2yYRyF3Xf28gUp3N8NxPF5'], 'uuids': ['20c42b36-20fe-4150-97ea-2830ed626f09'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.099378", "end": "2021-11-20 05:24:18.372202", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5SUP57dDziZP2yYRyF3Xf28gUp3N8NxPF5"], "labels": [], "masters": [], "uuids": ["20c42b36-20fe-4150-97ea-2830ed626f09"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:18.272824", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0965745 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0965745 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:18.454 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:24:18.456 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-11-20T05:24:18.476 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:18 +0000 (0:00:00.061) 0:05:15.229 ***** 2021-11-20T05:24:18.503 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-11-20T05:24:18.505 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:18.526 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:18 +0000 (0:00:00.049) 0:05:15.278 ***** 2021-11-20T05:24:18.553 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-11-20T05:24:18.555 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:18.576 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:18 +0000 (0:00:00.050) 0:05:15.329 ***** 2021-11-20T05:24:18.603 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-11-20T05:24:18.605 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:18.626 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:18 +0000 (0:00:00.049) 0:05:15.378 ***** 2021-11-20T05:24:18.673 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-11-20T05:24:18.674 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5STPq8Vl1uoBJAq12DCp5bgPWjFuqYGk1e'], 'uuids': ['069d4c61-0dc3-40af-9f4f-82e6ddeb42c7'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.098153", "end": "2021-11-20 05:24:18.651241", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5STPq8Vl1uoBJAq12DCp5bgPWjFuqYGk1e"], "labels": [], "masters": [], "uuids": ["069d4c61-0dc3-40af-9f4f-82e6ddeb42c7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:18.553088", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0953458 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0953458 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:18.686 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQFR', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQFR', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQFR', 'scsi-35000c500871a7ce9', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HQFR', 'wwn-0x5000c500871a7ce9'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4HQFR', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a7ce9', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQFR-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQFR-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQFR-part1', 'scsi-35000c500871a7ce9-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HQFR-part1', 'wwn-0x5000c500871a7ce9-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQFR", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQFR", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQFR", "scsi-35000c500871a7ce9", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HQFR", "wwn-0x5000c500871a7ce9"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQFR-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQFR-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-11-20T05:24:18.688 INFO:teuthology.task.ansible.out:Z1W4HQFR-part1", "scsi-35000c500871a7ce9-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HQFR-part1", "wwn-0x5000c500871a7ce9-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HQFR", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a7ce9"}}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:18.822 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-11-20T05:24:18.824 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-11-20T05:24:18.844 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:18 +0000 (0:00:00.218) 0:05:15.596 ***** 2021-11-20T05:24:18.964 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-11-20T05:24:18.966 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5S4VlicT1MXjDi5AilltMPww5X2kbhLE3w'], 'uuids': ['bfea6be8-cc5a-42e5-b893-1f0800dc443c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.098717", "end": "2021-11-20 05:24:18.936933", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-RiGWUjvv4PKz11LfLlqoIjtoAT83Bc5S4VlicT1MXjDi5AilltMPww5X2kbhLE3w"], "labels": [], "masters": [], "uuids": ["bfea6be8-cc5a-42e5-b893-1f0800dc443c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:24:18.838216", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0958841 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0958841 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:18.995 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:18 +0000 (0:00:00.151) 0:05:15.748 ***** 2021-11-20T05:24:19.241 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-11-20T05:24:19.243 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.052506", "end": "2021-11-20 05:24:19.217374", "rc": 0, "start": "2021-11-20 05:24:19.164868", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:19.263 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:19 +0000 (0:00:00.268) 0:05:16.016 ***** 2021-11-20T05:24:19.301 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-11-20T05:24:19.303 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-11-20T05:24:19.327 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:19 +0000 (0:00:00.063) 0:05:16.080 ***** 2021-11-20T05:24:19.501 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-11-20T05:24:19.504 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.048641", "end": "2021-11-20 05:24:19.478432", "rc": 0, "start": "2021-11-20 05:24:19.429791", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:19.525 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:19 +0000 (0:00:00.198) 0:05:16.278 ***** 2021-11-20T05:24:19.553 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-11-20T05:24:19.578 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:19 +0000 (0:00:00.052) 0:05:16.331 ***** 2021-11-20T05:24:19.605 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-11-20T05:24:19.607 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:19.630 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:19 +0000 (0:00:00.051) 0:05:16.383 ***** 2021-11-20T05:24:19.657 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-11-20T05:24:19.659 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:19.679 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:19 +0000 (0:00:00.049) 0:05:16.432 ***** 2021-11-20T05:24:19.707 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-11-20T05:24:19.709 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:19.734 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:19 +0000 (0:00:00.054) 0:05:16.487 ***** 2021-11-20T05:24:19.762 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-11-20T05:24:19.763 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:19.784 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:19 +0000 (0:00:00.049) 0:05:16.536 ***** 2021-11-20T05:24:19.981 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-11-20T05:24:19.983 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:24:20.005 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:20 +0000 (0:00:00.221) 0:05:16.758 ***** 2021-11-20T05:24:20.054 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-11-20T05:24:20.056 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140073984, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227084979, 'block_used': 13010129, 'inode_total': 61054976, 'inode_available': 60984950, 'inode_used': 70026, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227084979, "block_size": 4096, "block_total": 240095108, "block_used": 13010129, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984950, "inode_total": 61054976, "inode_used": 70026, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140073984, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:20.082 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:20 +0000 (0:00:00.077) 0:05:16.835 ***** 2021-11-20T05:24:20.110 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-11-20T05:24:20.112 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:20.137 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:20 +0000 (0:00:00.053) 0:05:16.889 ***** 2021-11-20T05:24:20.177 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:24:20.179 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:20.200 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:20 +0000 (0:00:00.063) 0:05:16.952 ***** 2021-11-20T05:24:20.227 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:24:20.229 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:20.250 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:20 +0000 (0:00:00.049) 0:05:17.002 ***** 2021-11-20T05:24:20.277 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-11-20T05:24:20.279 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:20.299 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:20 +0000 (0:00:00.049) 0:05:17.051 ***** 2021-11-20T05:24:20.326 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-11-20T05:24:20.328 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:20.348 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:20 +0000 (0:00:00.049) 0:05:17.101 ***** 2021-11-20T05:24:20.386 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-11-20T05:24:20.388 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:20.408 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:20 +0000 (0:00:00.059) 0:05:17.160 ***** 2021-11-20T05:24:20.540 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-11-20T05:24:20.542 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-11-20T05:24:20.563 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:20 +0000 (0:00:00.155) 0:05:17.316 ***** 2021-11-20T05:24:20.591 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-11-20T05:24:20.593 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:20.613 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:20 +0000 (0:00:00.049) 0:05:17.365 ***** 2021-11-20T05:24:20.639 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-11-20T05:24:20.641 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-11-20T05:24:20.661 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:20 +0000 (0:00:00.048) 0:05:17.413 ***** 2021-11-20T05:24:20.835 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-11-20T05:24:20.837 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-11-20T05:24:20.858 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:20 +0000 (0:00:00.197) 0:05:17.611 ***** 2021-11-20T05:24:20.865 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-11-20T05:24:20.867 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-11-20T05:24:20.893 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:20 +0000 (0:00:00.034) 0:05:17.645 ***** 2021-11-20T05:24:20.922 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:20.943 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:20 +0000 (0:00:00.050) 0:05:17.696 ***** 2021-11-20T05:24:20.970 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-11-20T05:24:20.972 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-11-20T05:24:20.993 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:20 +0000 (0:00:00.049) 0:05:17.745 ***** 2021-11-20T05:24:21.021 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:21.043 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:21 +0000 (0:00:00.050) 0:05:17.796 ***** 2021-11-20T05:24:21.237 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-11-20T05:24:21.238 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-11-20T05:24:21.258 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:21 +0000 (0:00:00.215) 0:05:18.011 ***** 2021-11-20T05:24:21.266 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-11-20T05:24:21.268 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:24:21.446 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi187.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-11-20T05:24:21.468 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:21 +0000 (0:00:00.209) 0:05:18.220 ***** 2021-11-20T05:24:21.495 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-11-20T05:24:21.497 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:21.518 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:21 +0000 (0:00:00.048) 0:05:18.269 ***** 2021-11-20T05:24:21.555 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi187.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-11-20T05:24:21.566 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-11-20T05:24:21.568 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-11-20T05:24:21.600 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:21 +0000 (0:00:00.083) 0:05:18.352 ***** 2021-11-20T05:24:21.617 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-11-20T05:24:21.619 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:24:21.640 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi187.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi187.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:21.651 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:21.653 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:21.663 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:21.687 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:21 +0000 (0:00:00.086) 0:05:18.439 ***** 2021-11-20T05:24:21.715 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:24:21.717 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:21.739 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:21 +0000 (0:00:00.052) 0:05:18.492 ***** 2021-11-20T05:24:21.767 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-11-20T05:24:21.769 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:21.790 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:21 +0000 (0:00:00.050) 0:05:18.542 ***** 2021-11-20T05:24:21.818 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:21.844 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:21 +0000 (0:00:00.052) 0:05:18.595 ***** 2021-11-20T05:24:21.871 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:21.893 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:21 +0000 (0:00:00.050) 0:05:18.645 ***** 2021-11-20T05:24:21.919 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-11-20T05:24:21.921 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:21.944 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:21 +0000 (0:00:00.050) 0:05:18.696 ***** 2021-11-20T05:24:21.960 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-11-20T05:24:21.962 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:24:21.972 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-11-20T05:24:21.974 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:21.997 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:21 +0000 (0:00:00.053) 0:05:18.749 ***** 2021-11-20T05:24:22.024 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-11-20T05:24:22.026 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:22.049 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:22 +0000 (0:00:00.052) 0:05:18.802 ***** 2021-11-20T05:24:22.078 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-11-20T05:24:22.079 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:22.101 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:22 +0000 (0:00:00.051) 0:05:18.854 ***** 2021-11-20T05:24:22.320 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-11-20T05:24:22.530 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi187.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005334", "end": "2021-11-20 05:24:22.354232", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-11-20 05:24:22.348898", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:22.566 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003074", "end": "2021-11-20 05:24:22.533528", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-11-20 05:24:22.530454", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:22.692 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-11-20T05:24:22.694 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:24:22.730 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:22 +0000 (0:00:00.629) 0:05:19.483 ***** 2021-11-20T05:24:22.740 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-11-20T05:24:22.742 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002769", "end": "2021-11-20 05:24:22.708562", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-11-20 05:24:22.705793", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:22.912 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003169", "end": "2021-11-20 05:24:22.883272", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-11-20 05:24:22.880103", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:22.940 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:22 +0000 (0:00:00.209) 0:05:19.693 ***** 2021-11-20T05:24:22.946 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-11-20T05:24:22.948 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-11-20T05:24:22.973 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:22 +0000 (0:00:00.032) 0:05:19.725 ***** 2021-11-20T05:24:22.981 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-11-20T05:24:23.014 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:23 +0000 (0:00:00.041) 0:05:19.767 ***** 2021-11-20T05:24:23.045 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-11-20T05:24:23.047 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:23.208 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-11-20T05:24:23.210 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T05:24:23.399 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T05:24:23.578 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T05:24:23.766 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T05:24:23.778 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:23.801 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:23 +0000 (0:00:00.786) 0:05:20.553 ***** 2021-11-20T05:24:23.839 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-11-20T05:24:23.841 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:23.861 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:23 +0000 (0:00:00.059) 0:05:20.613 ***** 2021-11-20T05:24:23.888 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-11-20T05:24:23.890 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:23.911 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:23 +0000 (0:00:00.050) 0:05:20.664 ***** 2021-11-20T05:24:24.117 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:24:24.139 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:24 +0000 (0:00:00.227) 0:05:20.891 ***** 2021-11-20T05:24:24.178 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi058.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-11-20T05:24:24.200 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:24 +0000 (0:00:00.060) 0:05:20.952 ***** 2021-11-20T05:24:24.428 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-11-20T05:24:24.430 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true} 2021-11-20T05:24:24.452 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:24 +0000 (0:00:00.252) 0:05:21.205 ***** 2021-11-20T05:24:24.782 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-11-20T05:24:24.783 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-11-20T05:24:24.803 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:24 +0000 (0:00:00.351) 0:05:21.556 ***** 2021-11-20T05:24:24.830 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-11-20T05:24:24.831 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:24.853 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:24 +0000 (0:00:00.049) 0:05:21.605 ***** 2021-11-20T05:24:24.880 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-11-20T05:24:24.882 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:24.903 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:24 +0000 (0:00:00.050) 0:05:21.656 ***** 2021-11-20T05:24:24.931 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:24.953 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:24 +0000 (0:00:00.049) 0:05:21.705 ***** 2021-11-20T05:24:24.980 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:25.002 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:24 +0000 (0:00:00.049) 0:05:21.755 ***** 2021-11-20T05:24:25.029 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:25.052 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:25 +0000 (0:00:00.049) 0:05:21.804 ***** 2021-11-20T05:24:25.079 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:25.100 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:25 +0000 (0:00:00.048) 0:05:21.853 ***** 2021-11-20T05:24:25.138 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-11-20T05:24:25.140 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:25.160 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:25 +0000 (0:00:00.059) 0:05:21.913 ***** 2021-11-20T05:24:25.197 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-11-20T05:24:25.199 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:25.219 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:25 +0000 (0:00:00.059) 0:05:21.972 ***** 2021-11-20T05:24:25.246 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-11-20T05:24:25.248 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:25.268 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:25 +0000 (0:00:00.048) 0:05:22.021 ***** 2021-11-20T05:24:25.295 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-11-20T05:24:25.298 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:25.317 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:25 +0000 (0:00:00.048) 0:05:22.070 ***** 2021-11-20T05:24:25.344 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-11-20T05:24:25.348 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:25.369 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:25 +0000 (0:00:00.052) 0:05:22.122 ***** 2021-11-20T05:24:25.397 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-11-20T05:24:25.399 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:25.418 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:25 +0000 (0:00:00.048) 0:05:22.171 ***** 2021-11-20T05:24:25.445 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:25.470 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:25 +0000 (0:00:00.051) 0:05:22.223 ***** 2021-11-20T05:24:25.497 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-11-20T05:24:25.499 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:25.518 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:25 +0000 (0:00:00.048) 0:05:22.271 ***** 2021-11-20T05:24:25.545 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:25.567 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:25 +0000 (0:00:00.048) 0:05:22.320 ***** 2021-11-20T05:24:26.062 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-11-20T05:24:26.083 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:26 +0000 (0:00:00.516) 0:05:22.836 ***** 2021-11-20T05:24:26.397 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-11-20T05:24:26.398 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-11-20 05:18:01 UTC", "ActiveEnterTimestampMonotonic": "15406336", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sntp.service systemd-journald.socket -.mount ntpd.service tmp.mount system.slice ntpdate.service basic.target systemd-tmpfiles-setup.service sysinit.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-11-20 05:18:00 UTC", "AssertTimestampMonotonic": "14023570", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-11-20 05:18:00 UTC", "ConditionTimestampMonotonic": "14023502", "ConfigurationDirectoryMode": "0755", "Conflicts": "syst 2021-11-20T05:24:26.400 INFO:teuthology.task.ansible.out:emd-timesyncd.service ntpd.service shutdown.target", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "899", "ExecMainStartTimestamp": "Sat 2021-11-20 05:18:01 UTC", "ExecMainStartTimestampMonotonic": "15337133", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-11-20 05:18:00 UTC", "InactiveExitTimestampMonotonic": "14054681", "InvocationID": "d6f2bcdfa25941789c4114b7e06ad992", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": 2021-11-20T05:24:26.402 INFO:teuthology.task.ansible.out:"infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127267", "LimitNPROCSoft": "127267", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127267", "LimitSIGPENDINGSoft": "127267", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "899", "MemoryAccounting": "yes", "MemoryCurrent": "1646592", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit 2021-11-20T05:24:26.403 INFO:teuthology.task.ansible.out:.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-11-20 05:18:01 UTC", "StateChangeTimestampMonotonic": "15406336", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203627", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-11-20 05:18:01 UTC", "WatchdogTimestampMonotonic": "15337404", "WatchdogUSec": "0"}} 2021-11-20T05:24:26.422 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:26 +0000 (0:00:00.339) 0:05:23.175 ***** 2021-11-20T05:24:26.450 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:24:26.452 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-11-20T05:24:26.473 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:26 +0000 (0:00:00.050) 0:05:23.225 ***** 2021-11-20T05:24:26.510 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:24:26.512 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-11-20T05:24:26.534 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:26 +0000 (0:00:00.060) 0:05:23.286 ***** 2021-11-20T05:24:26.560 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:26.582 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:26 +0000 (0:00:00.048) 0:05:23.335 ***** 2021-11-20T05:24:26.608 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-11-20T05:24:26.610 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:26.631 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:26 +0000 (0:00:00.048) 0:05:23.384 ***** 2021-11-20T05:24:26.658 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-11-20T05:24:26.660 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:26.680 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:26 +0000 (0:00:00.048) 0:05:23.433 ***** 2021-11-20T05:24:26.883 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-11-20T05:24:26.885 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-11-20T05:24:26.906 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:26 +0000 (0:00:00.225) 0:05:23.658 ***** 2021-11-20T05:24:27.399 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-11-20T05:24:27.401 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-11-20T05:24:27.424 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:27 +0000 (0:00:00.517) 0:05:24.176 ***** 2021-11-20T05:24:28.506 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-11-20T05:24:28.508 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-11-20T05:24:28.528 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:28 +0000 (0:00:01.104) 0:05:25.280 ***** 2021-11-20T05:24:28.566 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-11-20T05:24:28.568 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:28.588 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:28 +0000 (0:00:00.059) 0:05:25.340 ***** 2021-11-20T05:24:28.614 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-11-20T05:24:28.616 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-11-20T05:24:28.636 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:28 +0000 (0:00:00.048) 0:05:25.389 ***** 2021-11-20T05:24:28.830 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-11-20T05:24:28.832 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-11-20T05:24:28.852 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:28 +0000 (0:00:00.216) 0:05:25.605 ***** 2021-11-20T05:24:28.880 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-11-20T05:24:28.882 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:28.902 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:28 +0000 (0:00:00.049) 0:05:25.655 ***** 2021-11-20T05:24:28.950 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-11-20T05:24:28.952 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-11-20T05:24:28.973 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:28 +0000 (0:00:00.070) 0:05:25.726 ***** 2021-11-20T05:24:29.002 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-11-20T05:24:29.004 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:29.029 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:29 +0000 (0:00:00.056) 0:05:25.782 ***** 2021-11-20T05:24:29.235 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-11-20T05:24:29.237 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-11-20T05:24:29.266 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:29 +0000 (0:00:00.237) 0:05:26.019 ***** 2021-11-20T05:24:29.466 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-11-20T05:24:29.468 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-11-20T05:24:29.491 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:29 +0000 (0:00:00.224) 0:05:26.243 ***** 2021-11-20T05:24:29.519 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-11-20T05:24:29.520 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:29.544 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:29 +0000 (0:00:00.053) 0:05:26.297 ***** 2021-11-20T05:24:29.583 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-11-20T05:24:29.586 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-11-20T05:24:29.595 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2021-11-20T05:24:29.597 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi058.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi058.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-11-20T05:24:29.621 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:29 +0000 (0:00:00.076) 0:05:26.373 ***** 2021-11-20T05:24:29.659 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-11-20T05:24:29.661 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:29.671 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:29.672 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:29.682 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:29.684 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:29.706 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:29 +0000 (0:00:00.084) 0:05:26.458 ***** 2021-11-20T05:24:29.732 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:24:29.734 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:29.757 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:29 +0000 (0:00:00.051) 0:05:26.509 ***** 2021-11-20T05:24:29.784 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:29.806 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:29 +0000 (0:00:00.049) 0:05:26.559 ***** 2021-11-20T05:24:29.834 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:29.858 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:29 +0000 (0:00:00.051) 0:05:26.611 ***** 2021-11-20T05:24:29.885 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-11-20T05:24:29.887 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:29.908 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:29 +0000 (0:00:00.049) 0:05:26.660 ***** 2021-11-20T05:24:29.935 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:29.958 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:29 +0000 (0:00:00.050) 0:05:26.711 ***** 2021-11-20T05:24:29.985 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:30.010 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:30 +0000 (0:00:00.050) 0:05:26.762 ***** 2021-11-20T05:24:30.037 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-11-20T05:24:30.039 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:30.062 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:30 +0000 (0:00:00.053) 0:05:26.815 ***** 2021-11-20T05:24:30.090 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-11-20T05:24:30.092 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:30.113 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:30 +0000 (0:00:00.050) 0:05:26.865 ***** 2021-11-20T05:24:30.333 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi058.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005638", "end": "2021-11-20 05:24:30.300991", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-11-20 05:24:30.295353", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:30.521 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003255", "end": "2021-11-20 05:24:30.490777", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-11-20 05:24:30.487522", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:30.701 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.003021", "end": "2021-11-20 05:24:30.668773", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-11-20 05:24:30.665752", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:30.878 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003433", "end": "2021-11-20 05:24:30.850630", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-11-20 05:24:30.847197", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:24:30.905 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:30 +0000 (0:00:00.792) 0:05:27.657 ***** 2021-11-20T05:24:30.935 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-11-20T05:24:30.962 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:30 +0000 (0:00:00.057) 0:05:27.715 ***** 2021-11-20T05:24:30.991 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:31.061 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-11-20T05:24:31.084 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:31 +0000 (0:00:00.120) 0:05:27.835 ***** 2021-11-20T05:24:31.106 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:31 +0000 (0:00:00.023) 0:05:27.859 ***** 2021-11-20T05:24:31.743 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi058.front.sepia.ceph.com] 2021-11-20T05:24:31.800 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:31 +0000 (0:00:00.694) 0:05:28.553 ***** 2021-11-20T05:24:31.828 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T05:24:31.830 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:31.850 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:31 +0000 (0:00:00.049) 0:05:28.603 ***** 2021-11-20T05:24:31.879 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T05:24:31.881 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-11-20T05:24:31.907 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:31 +0000 (0:00:00.051) 0:05:28.654 ***** 2021-11-20T05:24:31.951 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-11-20T05:24:31.953 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-11-20T05:24:31.974 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:31 +0000 (0:00:00.071) 0:05:28.726 ***** 2021-11-20T05:24:32.773 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi187.front.sepia.ceph.com] 2021-11-20T05:24:32.822 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:32 +0000 (0:00:00.848) 0:05:29.575 ***** 2021-11-20T05:24:32.851 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T05:24:32.852 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:24:32.874 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:32 +0000 (0:00:00.051) 0:05:29.627 ***** 2021-11-20T05:24:32.913 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T05:24:32.915 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-11-20T05:24:32.936 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:32 +0000 (0:00:00.062) 0:05:29.689 ***** 2021-11-20T05:24:32.986 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-11-20T05:24:32.987 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-11-20T05:24:33.009 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:24:33 +0000 (0:00:00.072) 0:05:29.761 ***** 2021-11-20T05:25:16.352 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-11-20T05:25:16.354 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: containers-common-2:1-6.module_el8.6.0+954+963caf36.noarch", "Installed: podman-catatonit-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Installed: container-selinux-2:2.170.0-1.module_el8.6.0+954+963caf36.noarch", "Installed: podman-docker-1:3.4.1-3.module_el8.6.0+954+963caf36.noarch", "Installed: podman-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Removed: container-selinux-2:2.159.0-1.module_el8.5.0+733+9bb5dffa.noarch", "Removed: containers-common-1:1.2.2-4.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-catatonit-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-docker-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.noarch"]} 2021-11-20T05:25:16.374 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:16 +0000 (0:00:43.365) 0:06:13.126 ***** 2021-11-20T05:25:16.413 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T05:25:16.414 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-11-20T05:25:16.435 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:16 +0000 (0:00:00.061) 0:06:13.188 ***** 2021-11-20T05:25:16.462 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T05:25:16.464 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:25:16.484 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:16 +0000 (0:00:00.049) 0:06:13.237 ***** 2021-11-20T05:25:16.708 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-11-20T05:25:16.710 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:25:16.898 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:25:16.900 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-11-20T05:25:16.900 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: containers-common-2:1-6.module_el8.6.0+954+963caf36.noarch", "Installed: podman-catatonit-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Installed: container-selinux-2:2.170.0-1.module_el8.6.0+954+963caf36.noarch", "Installed: podman-docker-1:3.4.1-3.module_el8.6.0+954+963caf36.noarch", "Installed: podman-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Removed: container-selinux-2:2.159.0-1.module_el8.5.0+733+9bb5dffa.noarch", "Removed: containers-common-1:1.2.2-4.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-catatonit-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-docker-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.noarch"]} 2021-11-20T05:25:16.923 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:16 +0000 (0:00:00.438) 0:06:13.675 ***** 2021-11-20T05:25:16.946 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:16 +0000 (0:00:00.022) 0:06:13.698 ***** 2021-11-20T05:25:16.953 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T05:25:16.954 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-11-20T05:25:16.976 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:16 +0000 (0:00:00.030) 0:06:13.729 ***** 2021-11-20T05:25:17.004 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T05:25:17.006 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:25:17.032 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:17 +0000 (0:00:00.055) 0:06:13.785 ***** 2021-11-20T05:25:17.271 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-11-20T05:25:17.272 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:25:17.471 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:25:17.473 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-11-20T05:25:17.474 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637385917.017004-27989-238288733344242/source", "state": "file", "uid": 0} 2021-11-20T05:25:17.497 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:17 +0000 (0:00:00.464) 0:06:14.250 ***** 2021-11-20T05:25:17.958 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637385917.5065382-27989-68846113421067/source", "state": "file", "uid": 0} 2021-11-20T05:25:17.984 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:17 +0000 (0:00:00.486) 0:06:14.736 ***** 2021-11-20T05:25:18.056 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637385917.5652263-28070-114493149475329/source", "state": "file", "uid": 0} 2021-11-20T05:25:18.560 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637385918.0892448-28070-135767611620821/source", "state": "file", "uid": 0} 2021-11-20T05:25:18.585 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:18 +0000 (0:00:00.601) 0:06:15.338 ***** 2021-11-20T05:25:20.010 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** 2021-11-20T05:25:20.013 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:25:20.035 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:20 +0000 (0:00:01.448) 0:06:16.787 ***** 2021-11-20T05:25:20.753 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:25:20.775 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:20 +0000 (0:00:00.741) 0:06:17.528 ***** 2021-11-20T05:25:23.238 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-11-20T05:25:23.240 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-b75fjvyq-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-b75fjvyq-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-11-20T05:25:23.241 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-11-20T05:25:23.266 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:23 +0000 (0:00:02.490) 0:06:20.019 ***** 2021-11-20T05:25:23.305 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-11-20T05:25:23.308 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:25:23.335 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:23 +0000 (0:00:00.068) 0:06:20.088 ***** 2021-11-20T05:25:23.365 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-11-20T05:25:23.367 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:25:23.388 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:23 +0000 (0:00:00.052) 0:06:20.140 ***** 2021-11-20T05:25:23.729 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-11-20T05:25:23.731 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.146543", "end": "2021-11-20 05:25:23.709801", "rc": 0, "start": "2021-11-20 05:25:23.563258", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:25:23.751 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:23 +0000 (0:00:00.363) 0:06:20.503 ***** 2021-11-20T05:25:23.788 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-11-20T05:25:23.790 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:25:24.111 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-11-20T05:25:24.113 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-t_li25v1-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-t_li25v1-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-11-20T05:25:24.115 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-11-20T05:25:24.133 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:24 +0000 (0:00:00.381) 0:06:20.885 ***** 2021-11-20T05:25:24.160 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-11-20T05:25:24.162 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:25:24.182 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:24 +0000 (0:00:00.049) 0:06:20.934 ***** 2021-11-20T05:25:24.209 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-11-20T05:25:24.211 INFO:teuthology.task.ansible.out:skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:25:24.231 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:24 +0000 (0:00:00.049) 0:06:20.984 ***** 2021-11-20T05:25:24.593 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-11-20T05:25:24.595 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.153986", "end": "2021-11-20 05:25:24.571685", "rc": 0, "start": "2021-11-20 05:25:24.417699", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:25:24.615 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:24 +0000 (0:00:00.383) 0:06:21.367 ***** 2021-11-20T05:25:24.642 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* skipping: [smithi058.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:25:24.711 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-11-20T05:25:24.713 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-11-20T05:25:24.715 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-11-20T05:25:24.717 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-11-20T05:25:24.728 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-11-20T05:25:24.729 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:25:24 +0000 (0:00:00.113) 0:06:21.480 ***** 2021-11-20T05:25:25.383 INFO:teuthology.task.ansible.out:ok: [smithi058.front.sepia.ceph.com] 2021-11-20T05:25:25.410 INFO:teuthology.task.ansible.out:ok: [smithi187.front.sepia.ceph.com] 2021-11-20T05:25:25.430 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** Saturday 20 November 2021 05:25:25 +0000 (0:00:00.701) 0:06:22.182 ***** 2021-11-20T05:25:25.640 INFO:teuthology.task.ansible.out:changed: [smithi058.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-20T05:25:25.661 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-20T05:25:25.663 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* 2021-11-20T05:25:25.665 INFO:teuthology.task.ansible.out:smithi058.front.sepia.ceph.com : ok=138 changed=51 unreachable=0 failed=0 skipped=181 rescued=0 ignored=3 smithi187.front.sepia.ceph.com : ok=138 changed=51 unreachable=0 failed=0 skipped=181 rescued=0 ignored=3 Saturday 20 November 2021 05:25:25 +0000 (0:00:00.234) 0:06:22.416 ***** =============================================================================== Create all admin users with sudo access. ------------------------------- 61.09s container-host : Install container packages ---------------------------- 43.37s common : nrpe - Ensure SELinux policy is up to date -------------------- 34.46s users : Update authorized_keys using the keys repo --------------------- 29.86s Remove revoked users --------------------------------------------------- 15.37s testnode : Ensure ceph packages are not present. ----------------------- 10.27s common : nrpe - Remove SELinux policy package --------------------------- 9.80s common : nrpe - Load SELinux policy package ----------------------------- 9.06s testnode : Ensure ceph dependency packages are not present. ------------- 3.37s testnode : Install GPG keys --------------------------------------------- 2.61s testnode : Install GPG -------------------------------------------------- 2.58s container-host : Install registries-conf-ctl ---------------------------- 2.49s testnode : Make sure rpm dependencies are installed --------------------- 2.06s testnode : Remove packages ---------------------------------------------- 2.06s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.95s common : nrpe - Install SELinux dependencies ---------------------------- 1.94s testnode : Configure local mirrorlists ---------------------------------- 1.94s users : Update authorized_keys for each user with literal keys ---------- 1.83s Gathering Facts --------------------------------------------------------- 1.50s container-host : Ensure git is installed -------- 2021-11-20T05:25:25.667 INFO:teuthology.task.ansible.out:------------------------ 1.45s 2021-11-20T05:25:26.065 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi058.front.sepia.ceph.com'), Remote(name='ubuntu@smithi187.front.sepia.ceph.com')] 2021-11-20T05:25:26.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi058.front.sepia.ceph.com' 2021-11-20T05:25:26.070 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi058.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T05:25:26.144 DEBUG:teuthology.orchestra.run.smithi058:> true 2021-11-20T05:25:26.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2021-11-20T05:25:26.216 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T05:25:26.284 DEBUG:teuthology.orchestra.run.smithi187:> true 2021-11-20T05:25:26.351 INFO:teuthology.run_tasks:Running task clock... 2021-11-20T05:25:26.365 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-11-20T05:25:26.366 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-11-20T05:25:26.368 DEBUG:teuthology.orchestra.run.smithi058:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-20T05:25:26.370 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-11-20T05:25:26.371 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-20T05:25:26.400 INFO:teuthology.orchestra.run.smithi058.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-11-20T05:25:26.415 INFO:teuthology.orchestra.run.smithi058.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-11-20T05:25:26.422 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-11-20T05:25:26.437 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-11-20T05:25:26.449 INFO:teuthology.orchestra.run.smithi058.stderr:sudo: ntpd: command not found 2021-11-20T05:25:26.460 INFO:teuthology.orchestra.run.smithi058.stdout:506 Cannot talk to daemon 2021-11-20T05:25:26.470 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: ntpd: command not found 2021-11-20T05:25:26.475 INFO:teuthology.orchestra.run.smithi058.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-11-20T05:25:26.481 INFO:teuthology.orchestra.run.smithi187.stdout:506 Cannot talk to daemon 2021-11-20T05:25:26.488 INFO:teuthology.orchestra.run.smithi058.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-11-20T05:25:26.495 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-11-20T05:25:26.509 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-11-20T05:25:26.551 INFO:teuthology.orchestra.run.smithi058.stderr:bash: ntpq: command not found 2021-11-20T05:25:26.554 INFO:teuthology.orchestra.run.smithi058.stdout:210 Number of sources = 4 2021-11-20T05:25:26.554 INFO:teuthology.orchestra.run.smithi058.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-20T05:25:26.555 INFO:teuthology.orchestra.run.smithi058.stdout:=============================================================================== 2021-11-20T05:25:26.555 INFO:teuthology.orchestra.run.smithi058.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T05:25:26.556 INFO:teuthology.orchestra.run.smithi058.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T05:25:26.556 INFO:teuthology.orchestra.run.smithi058.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T05:25:26.557 INFO:teuthology.orchestra.run.smithi058.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T05:25:26.571 INFO:teuthology.orchestra.run.smithi187.stderr:bash: ntpq: command not found 2021-11-20T05:25:26.574 INFO:teuthology.orchestra.run.smithi187.stdout:210 Number of sources = 4 2021-11-20T05:25:26.574 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-20T05:25:26.575 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2021-11-20T05:25:26.575 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T05:25:26.575 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T05:25:26.575 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T05:25:26.576 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T05:25:26.576 INFO:teuthology.run_tasks:Running task pexec... 2021-11-20T05:25:26.587 INFO:teuthology.task.pexec:Executing custom commands... 2021-11-20T05:25:26.588 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi058.front.sepia.ceph.com 2021-11-20T05:25:26.588 DEBUG:teuthology.orchestra.run.smithi058:> TESTDIR=/home/ubuntu/cephtest bash -s 2021-11-20T05:25:26.589 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi187.front.sepia.ceph.com 2021-11-20T05:25:26.589 DEBUG:teuthology.orchestra.run.smithi187:> TESTDIR=/home/ubuntu/cephtest bash -s 2021-11-20T05:25:27.181 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:03 ago on Sat 20 Nov 2021 05:23:24 AM UTC. 2021-11-20T05:25:27.188 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:01:54 ago on Sat 20 Nov 2021 05:23:33 AM UTC. 2021-11-20T05:25:28.663 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:25:28.664 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:25:28.664 INFO:teuthology.orchestra.run.smithi058.stdout: Package Architecture Version Repository Size 2021-11-20T05:25:28.664 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:25:28.665 INFO:teuthology.orchestra.run.smithi058.stdout:Resetting modules: 2021-11-20T05:25:28.665 INFO:teuthology.orchestra.run.smithi058.stdout: container-tools 2021-11-20T05:25:28.665 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:25:28.665 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:25:28.665 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:25:28.666 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:25:28.670 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:25:28.671 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:25:28.671 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2021-11-20T05:25:28.671 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:25:28.672 INFO:teuthology.orchestra.run.smithi187.stdout:Resetting modules: 2021-11-20T05:25:28.672 INFO:teuthology.orchestra.run.smithi187.stdout: container-tools 2021-11-20T05:25:28.672 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:25:28.672 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:25:28.673 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:25:28.673 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:25:28.877 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:25:28.907 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:25:29.466 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:05 ago on Sat 20 Nov 2021 05:23:24 AM UTC. 2021-11-20T05:25:29.528 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:01:56 ago on Sat 20 Nov 2021 05:23:33 AM UTC. 2021-11-20T05:25:30.988 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:25:30.990 INFO:teuthology.orchestra.run.smithi187.stdout:===================================================================================================== 2021-11-20T05:25:30.991 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2021-11-20T05:25:30.991 INFO:teuthology.orchestra.run.smithi187.stdout:===================================================================================================== 2021-11-20T05:25:30.991 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrading: 2021-11-20T05:25:30.992 INFO:teuthology.orchestra.run.smithi187.stdout: conmon x86_64 2:2.0.30-1.module_el8.6.0+944+d413f95e CentOS-AppStream 54 k 2021-11-20T05:25:30.992 INFO:teuthology.orchestra.run.smithi187.stdout: containernetworking-plugins x86_64 1.0.1-1.module_el8.6.0+944+d413f95e CentOS-AppStream 19 M 2021-11-20T05:25:30.992 INFO:teuthology.orchestra.run.smithi187.stdout: criu x86_64 3.15-3.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 518 k 2021-11-20T05:25:30.992 INFO:teuthology.orchestra.run.smithi187.stdout: fuse-overlayfs x86_64 1.7.1-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 73 k 2021-11-20T05:25:30.993 INFO:teuthology.orchestra.run.smithi187.stdout: libslirp x86_64 4.4.0-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 70 k 2021-11-20T05:25:30.993 INFO:teuthology.orchestra.run.smithi187.stdout: runc x86_64 1.0.2-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 3.1 M 2021-11-20T05:25:30.993 INFO:teuthology.orchestra.run.smithi187.stdout: slirp4netns x86_64 1.1.8-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 51 k 2021-11-20T05:25:30.993 INFO:teuthology.orchestra.run.smithi187.stdout:Installing group/module packages: 2021-11-20T05:25:30.994 INFO:teuthology.orchestra.run.smithi187.stdout: buildah x86_64 1:1.23.1-2.module_el8.6.0+954+963caf36 CentOS-AppStream 7.9 M 2021-11-20T05:25:30.994 INFO:teuthology.orchestra.run.smithi187.stdout: cockpit-podman noarch 35-1.module_el8.6.0+944+d413f95e CentOS-AppStream 457 k 2021-11-20T05:25:30.995 INFO:teuthology.orchestra.run.smithi187.stdout: crun x86_64 1.2-1.module_el8.6.0+954+963caf36 CentOS-AppStream 193 k 2021-11-20T05:25:30.995 INFO:teuthology.orchestra.run.smithi187.stdout: python3-podman noarch 3.2.1-4.module_el8.6.0+954+963caf36 CentOS-AppStream 148 k 2021-11-20T05:25:30.995 INFO:teuthology.orchestra.run.smithi187.stdout: skopeo x86_64 2:1.5.0-2.module_el8.6.0+954+963caf36 CentOS-AppStream 6.7 M 2021-11-20T05:25:30.995 INFO:teuthology.orchestra.run.smithi187.stdout: toolbox x86_64 0.0.99.3-1.module_el8.5.0+877+1c30e0c9 CentOS-AppStream 2.3 M 2021-11-20T05:25:30.996 INFO:teuthology.orchestra.run.smithi187.stdout: udica noarch 0.2.6-2.module_el8.6.0+944+d413f95e CentOS-AppStream 48 k 2021-11-20T05:25:30.996 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2021-11-20T05:25:30.996 INFO:teuthology.orchestra.run.smithi187.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-11-20T05:25:30.996 INFO:teuthology.orchestra.run.smithi187.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-11-20T05:25:30.997 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-11-20T05:25:30.997 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 CentOS-AppStream 25 k 2021-11-20T05:25:30.997 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyxdg noarch 0.25-16.el8 CentOS-AppStream 94 k 2021-11-20T05:25:30.997 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-11-20T05:25:30.998 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-11-20T05:25:30.998 INFO:teuthology.orchestra.run.smithi187.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-11-20T05:25:30.998 INFO:teuthology.orchestra.run.smithi187.stdout:Installing module profiles: 2021-11-20T05:25:30.998 INFO:teuthology.orchestra.run.smithi187.stdout: container-tools/common 2021-11-20T05:25:30.998 INFO:teuthology.orchestra.run.smithi187.stdout:Enabling module streams: 2021-11-20T05:25:30.999 INFO:teuthology.orchestra.run.smithi187.stdout: container-tools rhel8 2021-11-20T05:25:30.999 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:25:30.999 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:25:30.999 INFO:teuthology.orchestra.run.smithi187.stdout:===================================================================================================== 2021-11-20T05:25:31.000 INFO:teuthology.orchestra.run.smithi187.stdout:Install 15 Packages 2021-11-20T05:25:31.000 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrade 7 Packages 2021-11-20T05:25:31.000 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:25:31.001 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 41 M 2021-11-20T05:25:31.001 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2021-11-20T05:25:31.020 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:25:31.024 INFO:teuthology.orchestra.run.smithi058.stdout:===================================================================================================== 2021-11-20T05:25:31.024 INFO:teuthology.orchestra.run.smithi058.stdout: Package Arch Version Repository Size 2021-11-20T05:25:31.024 INFO:teuthology.orchestra.run.smithi058.stdout:===================================================================================================== 2021-11-20T05:25:31.024 INFO:teuthology.orchestra.run.smithi058.stdout:Upgrading: 2021-11-20T05:25:31.025 INFO:teuthology.orchestra.run.smithi058.stdout: conmon x86_64 2:2.0.30-1.module_el8.6.0+944+d413f95e CentOS-AppStream 54 k 2021-11-20T05:25:31.025 INFO:teuthology.orchestra.run.smithi058.stdout: containernetworking-plugins x86_64 1.0.1-1.module_el8.6.0+944+d413f95e CentOS-AppStream 19 M 2021-11-20T05:25:31.025 INFO:teuthology.orchestra.run.smithi058.stdout: criu x86_64 3.15-3.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 518 k 2021-11-20T05:25:31.025 INFO:teuthology.orchestra.run.smithi058.stdout: fuse-overlayfs x86_64 1.7.1-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 73 k 2021-11-20T05:25:31.026 INFO:teuthology.orchestra.run.smithi058.stdout: libslirp x86_64 4.4.0-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 70 k 2021-11-20T05:25:31.026 INFO:teuthology.orchestra.run.smithi058.stdout: runc x86_64 1.0.2-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 3.1 M 2021-11-20T05:25:31.026 INFO:teuthology.orchestra.run.smithi058.stdout: slirp4netns x86_64 1.1.8-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 51 k 2021-11-20T05:25:31.026 INFO:teuthology.orchestra.run.smithi058.stdout:Installing group/module packages: 2021-11-20T05:25:31.027 INFO:teuthology.orchestra.run.smithi058.stdout: buildah x86_64 1:1.23.1-2.module_el8.6.0+954+963caf36 CentOS-AppStream 7.9 M 2021-11-20T05:25:31.027 INFO:teuthology.orchestra.run.smithi058.stdout: cockpit-podman noarch 35-1.module_el8.6.0+944+d413f95e CentOS-AppStream 457 k 2021-11-20T05:25:31.027 INFO:teuthology.orchestra.run.smithi058.stdout: crun x86_64 1.2-1.module_el8.6.0+954+963caf36 CentOS-AppStream 193 k 2021-11-20T05:25:31.027 INFO:teuthology.orchestra.run.smithi058.stdout: python3-podman noarch 3.2.1-4.module_el8.6.0+954+963caf36 CentOS-AppStream 148 k 2021-11-20T05:25:31.028 INFO:teuthology.orchestra.run.smithi058.stdout: skopeo x86_64 2:1.5.0-2.module_el8.6.0+954+963caf36 CentOS-AppStream 6.7 M 2021-11-20T05:25:31.028 INFO:teuthology.orchestra.run.smithi058.stdout: toolbox x86_64 0.0.99.3-1.module_el8.5.0+877+1c30e0c9 CentOS-AppStream 2.3 M 2021-11-20T05:25:31.028 INFO:teuthology.orchestra.run.smithi058.stdout: udica noarch 0.2.6-2.module_el8.6.0+944+d413f95e CentOS-AppStream 48 k 2021-11-20T05:25:31.028 INFO:teuthology.orchestra.run.smithi058.stdout:Installing dependencies: 2021-11-20T05:25:31.029 INFO:teuthology.orchestra.run.smithi058.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-11-20T05:25:31.029 INFO:teuthology.orchestra.run.smithi058.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-11-20T05:25:31.029 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-11-20T05:25:31.029 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 CentOS-AppStream 25 k 2021-11-20T05:25:31.030 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pyxdg noarch 0.25-16.el8 CentOS-AppStream 94 k 2021-11-20T05:25:31.030 INFO:teuthology.orchestra.run.smithi058.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-11-20T05:25:31.030 INFO:teuthology.orchestra.run.smithi058.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-11-20T05:25:31.030 INFO:teuthology.orchestra.run.smithi058.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-11-20T05:25:31.031 INFO:teuthology.orchestra.run.smithi058.stdout:Installing module profiles: 2021-11-20T05:25:31.031 INFO:teuthology.orchestra.run.smithi058.stdout: container-tools/common 2021-11-20T05:25:31.031 INFO:teuthology.orchestra.run.smithi058.stdout:Enabling module streams: 2021-11-20T05:25:31.031 INFO:teuthology.orchestra.run.smithi058.stdout: container-tools rhel8 2021-11-20T05:25:31.032 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:25:31.032 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:25:31.032 INFO:teuthology.orchestra.run.smithi058.stdout:===================================================================================================== 2021-11-20T05:25:31.032 INFO:teuthology.orchestra.run.smithi058.stdout:Install 15 Packages 2021-11-20T05:25:31.032 INFO:teuthology.orchestra.run.smithi058.stdout:Upgrade 7 Packages 2021-11-20T05:25:31.033 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:25:31.033 INFO:teuthology.orchestra.run.smithi058.stdout:Total download size: 41 M 2021-11-20T05:25:31.034 INFO:teuthology.orchestra.run.smithi058.stdout:Downloading Packages: 2021-11-20T05:25:31.047 INFO:teuthology.orchestra.run.smithi187.stdout:(1/22): crun-1.2-1.module_el8.6.0+954+963caf36. 3.6 MB/s | 193 kB 00:00 2021-11-20T05:25:31.049 INFO:teuthology.orchestra.run.smithi187.stdout:(2/22): cockpit-podman-35-1.module_el8.6.0+944+ 8.2 MB/s | 457 kB 00:00 2021-11-20T05:25:31.051 INFO:teuthology.orchestra.run.smithi187.stdout:(3/22): python3-pytoml-0.1.14-5.git7dea353.el8. 13 MB/s | 25 kB 00:00 2021-11-20T05:25:31.058 INFO:teuthology.orchestra.run.smithi187.stdout:(4/22): python3-pyxdg-0.25-16.el8.noarch.rpm 15 MB/s | 94 kB 00:00 2021-11-20T05:25:31.067 INFO:teuthology.orchestra.run.smithi187.stdout:(5/22): python3-podman-3.2.1-4.module_el8.6.0+9 7.3 MB/s | 148 kB 00:00 2021-11-20T05:25:31.090 INFO:teuthology.orchestra.run.smithi058.stdout:(1/22): crun-1.2-1.module_el8.6.0+954+963caf36. 3.0 MB/s | 193 kB 00:00 2021-11-20T05:25:31.101 INFO:teuthology.orchestra.run.smithi058.stdout:(2/22): python3-podman-3.2.1-4.module_el8.6.0+9 16 MB/s | 148 kB 00:00 2021-11-20T05:25:31.106 INFO:teuthology.orchestra.run.smithi058.stdout:(3/22): python3-pytoml-0.1.14-5.git7dea353.el8. 4.1 MB/s | 25 kB 00:00 2021-11-20T05:25:31.117 INFO:teuthology.orchestra.run.smithi058.stdout:(4/22): python3-pyxdg-0.25-16.el8.noarch.rpm 8.4 MB/s | 94 kB 00:00 2021-11-20T05:25:31.130 INFO:teuthology.orchestra.run.smithi058.stdout:(5/22): cockpit-podman-35-1.module_el8.6.0+944+ 4.4 MB/s | 457 kB 00:00 2021-11-20T05:25:31.206 INFO:teuthology.orchestra.run.smithi187.stdout:(6/22): buildah-1.23.1-2.module_el8.6.0+954+963 37 MB/s | 7.9 MB 00:00 2021-11-20T05:25:31.212 INFO:teuthology.orchestra.run.smithi187.stdout:(7/22): udica-0.2.6-2.module_el8.6.0+944+d413f9 7.7 MB/s | 48 kB 00:00 2021-11-20T05:25:31.219 INFO:teuthology.orchestra.run.smithi187.stdout:(8/22): yajl-2.1.0-10.el8.x86_64.rpm 6.3 MB/s | 41 kB 00:00 2021-11-20T05:25:31.227 INFO:teuthology.orchestra.run.smithi187.stdout:(9/22): python3-chardet-3.0.4-7.el8.noarch.rpm 23 MB/s | 195 kB 00:00 2021-11-20T05:25:31.236 INFO:teuthology.orchestra.run.smithi187.stdout:(10/22): python3-idna-2.5-5.el8.noarch.rpm 11 MB/s | 97 kB 00:00 2021-11-20T05:25:31.244 INFO:teuthology.orchestra.run.smithi187.stdout:(11/22): python3-pysocks-1.6.8-3.el8.noarch.rpm 4.7 MB/s | 34 kB 00:00 2021-11-20T05:25:31.255 INFO:teuthology.orchestra.run.smithi187.stdout:(12/22): python3-requests-2.20.0-2.1.el8_1.noar 10 MB/s | 123 kB 00:00 2021-11-20T05:25:31.269 INFO:teuthology.orchestra.run.smithi187.stdout:(13/22): python3-urllib3-1.24.2-5.el8.noarch.rp 13 MB/s | 177 kB 00:00 2021-11-20T05:25:31.278 INFO:teuthology.orchestra.run.smithi187.stdout:(14/22): conmon-2.0.30-1.module_el8.6.0+944+d41 5.6 MB/s | 54 kB 00:00 2021-11-20T05:25:31.346 INFO:teuthology.orchestra.run.smithi187.stdout:(15/22): toolbox-0.0.99.3-1.module_el8.5.0+877+ 8.1 MB/s | 2.3 MB 00:00 2021-11-20T05:25:31.405 INFO:teuthology.orchestra.run.smithi058.stdout:(6/22): skopeo-1.5.0-2.module_el8.6.0+954+963ca 23 MB/s | 6.7 MB 00:00 2021-11-20T05:25:31.421 INFO:teuthology.orchestra.run.smithi187.stdout:(16/22): criu-3.15-3.module_el8.6.0+926+8bef8ae 6.8 MB/s | 518 kB 00:00 2021-11-20T05:25:31.423 INFO:teuthology.orchestra.run.smithi058.stdout:(7/22): toolbox-0.0.99.3-1.module_el8.5.0+877+1 7.7 MB/s | 2.3 MB 00:00 2021-11-20T05:25:31.424 INFO:teuthology.orchestra.run.smithi058.stdout:(8/22): udica-0.2.6-2.module_el8.6.0+944+d413f9 2.5 MB/s | 48 kB 00:00 2021-11-20T05:25:31.433 INFO:teuthology.orchestra.run.smithi187.stdout:(17/22): fuse-overlayfs-1.7.1-1.module_el8.6.0+ 5.9 MB/s | 73 kB 00:00 2021-11-20T05:25:31.435 INFO:teuthology.orchestra.run.smithi058.stdout:(9/22): yajl-2.1.0-10.el8.x86_64.rpm 3.4 MB/s | 41 kB 00:00 2021-11-20T05:25:31.438 INFO:teuthology.orchestra.run.smithi058.stdout:(10/22): python3-chardet-3.0.4-7.el8.noarch.rpm 13 MB/s | 195 kB 00:00 2021-11-20T05:25:31.450 INFO:teuthology.orchestra.run.smithi058.stdout:(11/22): python3-idna-2.5-5.el8.noarch.rpm 6.2 MB/s | 97 kB 00:00 2021-11-20T05:25:31.451 INFO:teuthology.orchestra.run.smithi187.stdout:(18/22): libslirp-4.4.0-1.module_el8.6.0+926+8b 3.8 MB/s | 70 kB 00:00 2021-11-20T05:25:31.453 INFO:teuthology.orchestra.run.smithi058.stdout:(12/22): python3-pysocks-1.6.8-3.el8.noarch.rpm 2.3 MB/s | 34 kB 00:00 2021-11-20T05:25:31.471 INFO:teuthology.orchestra.run.smithi058.stdout:(13/22): python3-urllib3-1.24.2-5.el8.noarch.rp 9.7 MB/s | 177 kB 00:00 2021-11-20T05:25:31.481 INFO:teuthology.orchestra.run.smithi058.stdout:(14/22): python3-requests-2.20.0-2.1.el8_1.noar 4.1 MB/s | 123 kB 00:00 2021-11-20T05:25:31.486 INFO:teuthology.orchestra.run.smithi058.stdout:(15/22): conmon-2.0.30-1.module_el8.6.0+944+d41 3.7 MB/s | 54 kB 00:00 2021-11-20T05:25:31.528 INFO:teuthology.orchestra.run.smithi058.stdout:(16/22): criu-3.15-3.module_el8.6.0+926+8bef8ae 12 MB/s | 518 kB 00:00 2021-11-20T05:25:31.545 INFO:teuthology.orchestra.run.smithi058.stdout:(17/22): fuse-overlayfs-1.7.1-1.module_el8.6.0+ 4.2 MB/s | 73 kB 00:00 2021-11-20T05:25:31.565 INFO:teuthology.orchestra.run.smithi058.stdout:(18/22): libslirp-4.4.0-1.module_el8.6.0+926+8b 3.5 MB/s | 70 kB 00:00 2021-11-20T05:25:31.605 INFO:teuthology.orchestra.run.smithi187.stdout:(19/22): skopeo-1.5.0-2.module_el8.6.0+954+963c 12 MB/s | 6.7 MB 00:00 2021-11-20T05:25:31.628 INFO:teuthology.orchestra.run.smithi187.stdout:(20/22): slirp4netns-1.1.8-1.module_el8.6.0+926 2.2 MB/s | 51 kB 00:00 2021-11-20T05:25:31.729 INFO:teuthology.orchestra.run.smithi058.stdout:(19/22): runc-1.0.2-1.module_el8.6.0+926+8bef8a 19 MB/s | 3.1 MB 00:00 2021-11-20T05:25:31.745 INFO:teuthology.orchestra.run.smithi058.stdout:(20/22): slirp4netns-1.1.8-1.module_el8.6.0+926 3.2 MB/s | 51 kB 00:00 2021-11-20T05:25:31.872 INFO:teuthology.orchestra.run.smithi058.stdout:(21/22): buildah-1.23.1-2.module_el8.6.0+954+96 9.4 MB/s | 7.9 MB 00:00 2021-11-20T05:25:31.934 INFO:teuthology.orchestra.run.smithi187.stdout:(21/22): containernetworking-plugins-1.0.1-1.mo 29 MB/s | 19 MB 00:00 2021-11-20T05:25:31.942 INFO:teuthology.orchestra.run.smithi187.stdout:(22/22): runc-1.0.2-1.module_el8.6.0+926+8bef8a 6.3 MB/s | 3.1 MB 00:00 2021-11-20T05:25:31.945 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:25:31.946 INFO:teuthology.orchestra.run.smithi187.stdout:Total 43 MB/s | 41 MB 00:00 2021-11-20T05:25:31.979 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2021-11-20T05:25:32.006 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2021-11-20T05:25:32.006 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2021-11-20T05:25:32.147 INFO:teuthology.orchestra.run.smithi058.stdout:(22/22): containernetworking-plugins-1.0.1-1.mo 29 MB/s | 19 MB 00:00 2021-11-20T05:25:32.150 INFO:teuthology.orchestra.run.smithi058.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:25:32.151 INFO:teuthology.orchestra.run.smithi058.stdout:Total 37 MB/s | 41 MB 00:01 2021-11-20T05:25:32.187 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction check 2021-11-20T05:25:32.217 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction check succeeded. 2021-11-20T05:25:32.218 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction test 2021-11-20T05:25:32.345 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2021-11-20T05:25:32.348 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2021-11-20T05:25:32.578 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction test succeeded. 2021-11-20T05:25:32.582 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction 2021-11-20T05:25:33.496 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2021-11-20T05:25:33.659 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 1/29 2021-11-20T05:25:33.734 INFO:teuthology.orchestra.run.smithi058.stdout: Preparing : 1/1 2021-11-20T05:25:33.855 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 2/29 2021-11-20T05:25:33.903 INFO:teuthology.orchestra.run.smithi058.stdout: Upgrading : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 1/29 2021-11-20T05:25:34.068 INFO:teuthology.orchestra.run.smithi058.stdout: Upgrading : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 2/29 2021-11-20T05:25:34.416 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 3/29 2021-11-20T05:25:34.598 INFO:teuthology.orchestra.run.smithi058.stdout: Upgrading : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 3/29 2021-11-20T05:25:34.619 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 4/29 2021-11-20T05:25:34.788 INFO:teuthology.orchestra.run.smithi058.stdout: Upgrading : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 4/29 2021-11-20T05:25:34.832 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 5/29 2021-11-20T05:25:35.022 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 5/29 2021-11-20T05:25:35.056 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 6/29 2021-11-20T05:25:35.227 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 6/29 2021-11-20T05:25:35.237 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-idna-2.5-5.el8.noarch 7/29 2021-11-20T05:25:35.420 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-idna-2.5-5.el8.noarch 7/29 2021-11-20T05:25:35.450 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 8/29 2021-11-20T05:25:35.617 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 9/29 2021-11-20T05:25:35.621 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 8/29 2021-11-20T05:25:35.784 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-20T05:25:35.786 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 9/29 2021-11-20T05:25:35.929 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 11/29 2021-11-20T05:25:35.955 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-20T05:25:36.095 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 11/29 2021-11-20T05:25:36.154 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 12/29 2021-11-20T05:25:36.312 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 12/29 2021-11-20T05:25:36.352 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 13/29 2021-11-20T05:25:36.506 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 13/29 2021-11-20T05:25:37.157 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 14/29 2021-11-20T05:25:37.383 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 14/29 2021-11-20T05:25:37.397 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 15/29 2021-11-20T05:25:37.577 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-20T05:25:37.600 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 15/29 2021-11-20T05:25:37.754 INFO:teuthology.orchestra.run.smithi058.stdout: Upgrading : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-20T05:25:39.182 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-20T05:25:39.392 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : containernetworking-plugins-1.0.1-1.module_el8.6.0 17/29 2021-11-20T05:25:39.467 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-20T05:25:39.557 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 18/29 2021-11-20T05:25:39.685 INFO:teuthology.orchestra.run.smithi058.stdout: Upgrading : containernetworking-plugins-1.0.1-1.module_el8.6.0 17/29 2021-11-20T05:25:39.902 INFO:teuthology.orchestra.run.smithi058.stdout: Upgrading : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 18/29 2021-11-20T05:25:39.940 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 19/29 2021-11-20T05:25:40.315 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 19/29 2021-11-20T05:25:40.659 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 20/29 2021-11-20T05:25:40.848 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 21/29 2021-11-20T05:25:41.027 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 22/29 2021-11-20T05:25:41.084 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 20/29 2021-11-20T05:25:41.213 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 23/29 2021-11-20T05:25:41.276 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 21/29 2021-11-20T05:25:41.369 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 24/29 2021-11-20T05:25:41.433 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 22/29 2021-11-20T05:25:41.563 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 25/29 2021-11-20T05:25:41.605 INFO:teuthology.orchestra.run.smithi058.stdout: Cleanup : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 23/29 2021-11-20T05:25:41.702 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 26/29 2021-11-20T05:25:41.780 INFO:teuthology.orchestra.run.smithi058.stdout: Cleanup : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 24/29 2021-11-20T05:25:41.878 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 27/29 2021-11-20T05:25:41.917 INFO:teuthology.orchestra.run.smithi058.stdout: Cleanup : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 25/29 2021-11-20T05:25:42.077 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : containernetworking-plugins-0.9.1-1.module_el8.5.0 28/29 2021-11-20T05:25:42.096 INFO:teuthology.orchestra.run.smithi058.stdout: Cleanup : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 26/29 2021-11-20T05:25:42.225 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-20T05:25:42.281 INFO:teuthology.orchestra.run.smithi058.stdout: Cleanup : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 27/29 2021-11-20T05:25:42.479 INFO:teuthology.orchestra.run.smithi058.stdout: Cleanup : containernetworking-plugins-0.9.1-1.module_el8.5.0 28/29 2021-11-20T05:25:42.620 INFO:teuthology.orchestra.run.smithi058.stdout: Cleanup : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-20T05:25:42.653 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-20T05:25:42.653 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 1/29 2021-11-20T05:25:42.654 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 2/29 2021-11-20T05:25:42.654 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 3/29 2021-11-20T05:25:42.654 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/29 2021-11-20T05:25:42.654 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 5/29 2021-11-20T05:25:42.655 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 6/29 2021-11-20T05:25:42.655 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 7/29 2021-11-20T05:25:42.655 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 8/29 2021-11-20T05:25:42.655 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 9/29 2021-11-20T05:25:42.656 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-20T05:25:42.656 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 11/29 2021-11-20T05:25:42.656 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-idna-2.5-5.el8.noarch 12/29 2021-11-20T05:25:42.656 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 13/29 2021-11-20T05:25:42.656 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 14/29 2021-11-20T05:25:42.657 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 15/29 2021-11-20T05:25:42.657 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 16/29 2021-11-20T05:25:42.657 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 17/29 2021-11-20T05:25:42.658 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : containernetworking-plugins-1.0.1-1.module_el8.6.0 18/29 2021-11-20T05:25:42.658 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : containernetworking-plugins-0.9.1-1.module_el8.5.0 19/29 2021-11-20T05:25:42.658 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 20/29 2021-11-20T05:25:42.658 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 21/29 2021-11-20T05:25:42.659 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 22/29 2021-11-20T05:25:42.659 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 23/29 2021-11-20T05:25:42.659 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 24/29 2021-11-20T05:25:42.659 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 25/29 2021-11-20T05:25:42.660 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 26/29 2021-11-20T05:25:42.660 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 27/29 2021-11-20T05:25:42.661 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 28/29 2021-11-20T05:25:43.049 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-20T05:25:43.050 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 1/29 2021-11-20T05:25:43.050 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 2/29 2021-11-20T05:25:43.050 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 3/29 2021-11-20T05:25:43.051 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/29 2021-11-20T05:25:43.051 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 5/29 2021-11-20T05:25:43.051 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 6/29 2021-11-20T05:25:43.051 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 7/29 2021-11-20T05:25:43.052 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 8/29 2021-11-20T05:25:43.052 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 9/29 2021-11-20T05:25:43.052 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-20T05:25:43.052 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 11/29 2021-11-20T05:25:43.053 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-idna-2.5-5.el8.noarch 12/29 2021-11-20T05:25:43.053 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 13/29 2021-11-20T05:25:43.053 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 14/29 2021-11-20T05:25:43.053 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 15/29 2021-11-20T05:25:43.053 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 16/29 2021-11-20T05:25:43.054 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 17/29 2021-11-20T05:25:43.054 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : containernetworking-plugins-1.0.1-1.module_el8.6.0 18/29 2021-11-20T05:25:43.054 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : containernetworking-plugins-0.9.1-1.module_el8.5.0 19/29 2021-11-20T05:25:43.055 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 20/29 2021-11-20T05:25:43.055 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 21/29 2021-11-20T05:25:43.055 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 22/29 2021-11-20T05:25:43.056 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 23/29 2021-11-20T05:25:43.056 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 24/29 2021-11-20T05:25:43.056 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 25/29 2021-11-20T05:25:43.056 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 26/29 2021-11-20T05:25:43.056 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 27/29 2021-11-20T05:25:43.057 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 28/29 2021-11-20T05:25:43.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 29/29 2021-11-20T05:25:43.749 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:25:43.749 INFO:teuthology.orchestra.run.smithi187.stdout:Upgraded: 2021-11-20T05:25:43.749 INFO:teuthology.orchestra.run.smithi187.stdout: conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-20T05:25:43.750 INFO:teuthology.orchestra.run.smithi187.stdout: containernetworking-plugins-1.0.1-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-20T05:25:43.750 INFO:teuthology.orchestra.run.smithi187.stdout: criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:25:43.750 INFO:teuthology.orchestra.run.smithi187.stdout: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:25:43.750 INFO:teuthology.orchestra.run.smithi187.stdout: libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:25:43.751 INFO:teuthology.orchestra.run.smithi187.stdout: runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:25:43.751 INFO:teuthology.orchestra.run.smithi187.stdout: slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:25:43.751 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2021-11-20T05:25:43.753 INFO:teuthology.orchestra.run.smithi187.stdout: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86_64 2021-11-20T05:25:43.753 INFO:teuthology.orchestra.run.smithi187.stdout: cockpit-podman-35-1.module_el8.6.0+944+d413f95e.noarch 2021-11-20T05:25:43.753 INFO:teuthology.orchestra.run.smithi187.stdout: crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 2021-11-20T05:25:43.753 INFO:teuthology.orchestra.run.smithi187.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-11-20T05:25:43.754 INFO:teuthology.orchestra.run.smithi187.stdout: python3-idna-2.5-5.el8.noarch 2021-11-20T05:25:43.754 INFO:teuthology.orchestra.run.smithi187.stdout: python3-podman-3.2.1-4.module_el8.6.0+954+963caf36.noarch 2021-11-20T05:25:43.754 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-11-20T05:25:43.754 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2021-11-20T05:25:43.754 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyxdg-0.25-16.el8.noarch 2021-11-20T05:25:43.755 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-11-20T05:25:43.755 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-11-20T05:25:43.755 INFO:teuthology.orchestra.run.smithi187.stdout: skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_64 2021-11-20T05:25:43.758 INFO:teuthology.orchestra.run.smithi187.stdout: toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86_64 2021-11-20T05:25:43.758 INFO:teuthology.orchestra.run.smithi187.stdout: udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 2021-11-20T05:25:43.758 INFO:teuthology.orchestra.run.smithi187.stdout: yajl-2.1.0-10.el8.x86_64 2021-11-20T05:25:43.759 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:25:43.759 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:25:43.796 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 29/29 2021-11-20T05:25:43.797 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:25:43.797 INFO:teuthology.orchestra.run.smithi058.stdout:Upgraded: 2021-11-20T05:25:43.797 INFO:teuthology.orchestra.run.smithi058.stdout: conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-20T05:25:43.797 INFO:teuthology.orchestra.run.smithi058.stdout: containernetworking-plugins-1.0.1-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-20T05:25:43.797 INFO:teuthology.orchestra.run.smithi058.stdout: criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:25:43.798 INFO:teuthology.orchestra.run.smithi058.stdout: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:25:43.798 INFO:teuthology.orchestra.run.smithi058.stdout: libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:25:43.798 INFO:teuthology.orchestra.run.smithi058.stdout: runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:25:43.798 INFO:teuthology.orchestra.run.smithi058.stdout: slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:25:43.799 INFO:teuthology.orchestra.run.smithi058.stdout:Installed: 2021-11-20T05:25:43.799 INFO:teuthology.orchestra.run.smithi058.stdout: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86_64 2021-11-20T05:25:43.799 INFO:teuthology.orchestra.run.smithi058.stdout: cockpit-podman-35-1.module_el8.6.0+944+d413f95e.noarch 2021-11-20T05:25:43.799 INFO:teuthology.orchestra.run.smithi058.stdout: crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 2021-11-20T05:25:43.800 INFO:teuthology.orchestra.run.smithi058.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-11-20T05:25:43.800 INFO:teuthology.orchestra.run.smithi058.stdout: python3-idna-2.5-5.el8.noarch 2021-11-20T05:25:43.800 INFO:teuthology.orchestra.run.smithi058.stdout: python3-podman-3.2.1-4.module_el8.6.0+954+963caf36.noarch 2021-11-20T05:25:43.800 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-11-20T05:25:43.800 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2021-11-20T05:25:43.801 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pyxdg-0.25-16.el8.noarch 2021-11-20T05:25:43.801 INFO:teuthology.orchestra.run.smithi058.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-11-20T05:25:43.801 INFO:teuthology.orchestra.run.smithi058.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-11-20T05:25:43.801 INFO:teuthology.orchestra.run.smithi058.stdout: skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_64 2021-11-20T05:25:43.802 INFO:teuthology.orchestra.run.smithi058.stdout: toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86_64 2021-11-20T05:25:43.802 INFO:teuthology.orchestra.run.smithi058.stdout: udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 2021-11-20T05:25:43.803 INFO:teuthology.orchestra.run.smithi058.stdout: yajl-2.1.0-10.el8.x86_64 2021-11-20T05:25:43.803 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:25:43.803 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:25:44.003 DEBUG:teuthology.parallel:result is None 2021-11-20T05:25:44.050 DEBUG:teuthology.parallel:result is None 2021-11-20T05:25:44.050 INFO:teuthology.run_tasks:Running task install... 2021-11-20T05:25:44.063 DEBUG:teuthology.task.install:project ceph 2021-11-20T05:25:44.064 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa'}} 2021-11-20T05:25:44.064 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa'} 2021-11-20T05:25:44.065 INFO:teuthology.task.install:Using flavor: default 2021-11-20T05:25:44.072 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-11-20T05:25:44.073 INFO:teuthology.task.install:extra packages: [] 2021-11-20T05:25:44.073 DEBUG:teuthology.orchestra.run.smithi058:> sudo lsb_release -is 2021-11-20T05:25:44.108 INFO:teuthology.orchestra.run.smithi058.stdout:CentOSStream 2021-11-20T05:25:44.109 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T05:25:44.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo lsb_release -is 2021-11-20T05:25:44.111 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa', 'tag': 'v16.2.4', 'wait_for_package': False} 2021-11-20T05:25:44.112 DEBUG:teuthology.orchestra.run.smithi058:> sudo lsb_release -is 2021-11-20T05:25:44.145 INFO:teuthology.orchestra.run.smithi187.stdout:CentOSStream 2021-11-20T05:25:44.147 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T05:25:44.148 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa', 'tag': 'v16.2.4', 'wait_for_package': False} 2021-11-20T05:25:44.148 DEBUG:teuthology.orchestra.run.smithi187:> sudo lsb_release -is 2021-11-20T05:25:44.184 INFO:teuthology.orchestra.run.smithi058.stdout:CentOSStream 2021-11-20T05:25:44.185 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T05:25:44.186 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T05:25:44.186 INFO:teuthology.packaging:ref: None 2021-11-20T05:25:44.186 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T05:25:44.187 INFO:teuthology.packaging:branch: None 2021-11-20T05:25:44.187 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T05:25:44.402 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T05:25:44.404 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T05:25:44.442 INFO:teuthology.orchestra.run.smithi187.stdout:CentOSStream 2021-11-20T05:25:44.444 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T05:25:44.444 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T05:25:44.445 INFO:teuthology.packaging:ref: None 2021-11-20T05:25:44.445 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T05:25:44.446 INFO:teuthology.packaging:branch: None 2021-11-20T05:25:44.446 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T05:25:44.814 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T05:25:44.815 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T05:25:44.989 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2021-11-20T05:25:44.989 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2021-11-20T05:25:45.025 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-11-20T05:25:45.026 DEBUG:teuthology.orchestra.run.smithi058:> set -ex 2021-11-20T05:25:45.026 DEBUG:teuthology.orchestra.run.smithi058:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-11-20T05:25:45.030 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2021-11-20T05:25:45.030 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2021-11-20T05:25:45.055 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-11-20T05:25:45.056 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2021-11-20T05:25:45.056 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-11-20T05:25:45.064 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-11-20T05:25:45.065 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T05:25:45.065 INFO:teuthology.packaging:ref: None 2021-11-20T05:25:45.066 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T05:25:45.066 INFO:teuthology.packaging:branch: None 2021-11-20T05:25:45.067 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T05:25:45.068 DEBUG:teuthology.orchestra.run.smithi058:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-11-20T05:25:45.083 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-11-20T05:25:45.084 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T05:25:45.084 INFO:teuthology.packaging:ref: None 2021-11-20T05:25:45.085 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T05:25:45.085 INFO:teuthology.packaging:branch: None 2021-11-20T05:25:45.086 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T05:25:45.086 DEBUG:teuthology.orchestra.run.smithi187:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-11-20T05:25:45.136 DEBUG:teuthology.orchestra.run.smithi058:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-11-20T05:25:45.151 DEBUG:teuthology.orchestra.run.smithi187:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-11-20T05:25:45.218 DEBUG:teuthology.orchestra.run.smithi058:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-11-20T05:25:45.227 DEBUG:teuthology.orchestra.run.smithi187:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-11-20T05:25:45.285 INFO:teuthology.orchestra.run.smithi058.stdout:check_obsoletes = 1 2021-11-20T05:25:45.287 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum clean all 2021-11-20T05:25:45.290 INFO:teuthology.orchestra.run.smithi187.stdout:check_obsoletes = 1 2021-11-20T05:25:45.292 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum clean all 2021-11-20T05:25:45.527 INFO:teuthology.orchestra.run.smithi187.stdout:75 files removed 2021-11-20T05:25:45.539 INFO:teuthology.orchestra.run.smithi058.stdout:75 files removed 2021-11-20T05:25:45.548 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-radosgw 2021-11-20T05:25:45.559 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install ceph-radosgw 2021-11-20T05:25:45.955 INFO:teuthology.orchestra.run.smithi058.stdout:ceph packages for x86_64 1.9 MB/s | 74 kB 00:00 2021-11-20T05:25:45.958 INFO:teuthology.orchestra.run.smithi187.stdout:ceph packages for x86_64 1.0 MB/s | 74 kB 00:00 2021-11-20T05:25:46.095 INFO:teuthology.orchestra.run.smithi058.stdout:ceph noarch packages 150 kB/s | 15 kB 00:00 2021-11-20T05:25:46.096 INFO:teuthology.orchestra.run.smithi187.stdout:ceph noarch packages 152 kB/s | 15 kB 00:00 2021-11-20T05:25:46.178 INFO:teuthology.orchestra.run.smithi187.stdout:ceph source packages 32 kB/s | 1.7 kB 00:00 2021-11-20T05:25:46.179 INFO:teuthology.orchestra.run.smithi058.stdout:ceph source packages 32 kB/s | 1.7 kB 00:00 2021-11-20T05:25:46.674 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-8 - AppStream 36 MB/s | 17 MB 00:00 2021-11-20T05:25:46.717 INFO:teuthology.orchestra.run.smithi058.stdout:CentOS-8 - AppStream 34 MB/s | 17 MB 00:00 2021-11-20T05:25:49.837 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-8 - Base 52 MB/s | 15 MB 00:00 2021-11-20T05:25:50.075 INFO:teuthology.orchestra.run.smithi058.stdout:CentOS-8 - Base 50 MB/s | 15 MB 00:00 2021-11-20T05:25:52.100 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-8 - Extras 215 kB/s | 16 kB 00:00 2021-11-20T05:25:52.320 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-8 - PowerTools 30 MB/s | 3.6 MB 00:00 2021-11-20T05:25:52.410 INFO:teuthology.orchestra.run.smithi058.stdout:CentOS-8 - Extras 229 kB/s | 16 kB 00:00 2021-11-20T05:25:52.633 INFO:teuthology.orchestra.run.smithi058.stdout:CentOS-8 - PowerTools 32 MB/s | 3.6 MB 00:00 2021-11-20T05:25:55.328 INFO:teuthology.orchestra.run.smithi058.stdout:Copr repo for python3-asyncssh owned by ceph 1.8 kB/s | 3.5 kB 00:01 2021-11-20T05:25:55.349 INFO:teuthology.orchestra.run.smithi187.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2021-11-20T05:25:56.700 INFO:teuthology.orchestra.run.smithi187.stdout:Apache Arrow 148 kB/s | 184 kB 00:01 2021-11-20T05:25:56.703 INFO:teuthology.orchestra.run.smithi058.stdout:Apache Arrow 146 kB/s | 184 kB 00:01 2021-11-20T05:25:57.709 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 12 MB/s | 11 MB 00:00 2021-11-20T05:25:57.739 INFO:teuthology.orchestra.run.smithi058.stdout:Extra Packages for Enterprise Linux 12 MB/s | 11 MB 00:00 2021-11-20T05:26:00.099 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2021-11-20T05:26:00.365 INFO:teuthology.orchestra.run.smithi058.stdout:lab-extras 4.7 MB/s | 24 kB 00:00 2021-11-20T05:26:02.869 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:26:02.871 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:26:02.872 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2021-11-20T05:26:02.872 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:26:02.872 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2021-11-20T05:26:02.872 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2021-11-20T05:26:02.872 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrading: 2021-11-20T05:26:02.873 INFO:teuthology.orchestra.run.smithi187.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-20T05:26:02.873 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2021-11-20T05:26:02.873 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2021-11-20T05:26:02.873 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2021-11-20T05:26:02.874 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2021-11-20T05:26:02.874 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2021-11-20T05:26:02.874 INFO:teuthology.orchestra.run.smithi187.stdout: daxctl-libs x86_64 71.1-2.el8 CentOS-Base 41 k 2021-11-20T05:26:02.874 INFO:teuthology.orchestra.run.smithi187.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-11-20T05:26:02.874 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-11-20T05:26:02.875 INFO:teuthology.orchestra.run.smithi187.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-11-20T05:26:02.875 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2021-11-20T05:26:02.877 INFO:teuthology.orchestra.run.smithi187.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-11-20T05:26:02.877 INFO:teuthology.orchestra.run.smithi187.stdout: libpmemobj x86_64 1.6.1-1.el8 CentOS-AppStream 145 k 2021-11-20T05:26:02.878 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-11-20T05:26:02.878 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2021-11-20T05:26:02.878 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2021-11-20T05:26:02.878 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2021-11-20T05:26:02.879 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-11-20T05:26:02.879 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-11-20T05:26:02.879 INFO:teuthology.orchestra.run.smithi187.stdout: ndctl-libs x86_64 71.1-2.el8 CentOS-Base 78 k 2021-11-20T05:26:02.879 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2021-11-20T05:26:02.879 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2021-11-20T05:26:02.880 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2021-11-20T05:26:02.880 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-11-20T05:26:02.880 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2021-11-20T05:26:02.880 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2021-11-20T05:26:02.880 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2021-11-20T05:26:02.882 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:26:02.882 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:26:02.882 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:26:02.883 INFO:teuthology.orchestra.run.smithi187.stdout:Install 25 Packages 2021-11-20T05:26:02.883 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrade 2 Packages 2021-11-20T05:26:02.883 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:26:02.884 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 59 M 2021-11-20T05:26:02.884 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2021-11-20T05:26:03.031 INFO:teuthology.orchestra.run.smithi187.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 39 MB/s | 6.0 MB 00:00 2021-11-20T05:26:03.037 INFO:teuthology.orchestra.run.smithi187.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 3.9 MB/s | 25 kB 00:00 2021-11-20T05:26:03.067 INFO:teuthology.orchestra.run.smithi187.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 26 MB/s | 801 kB 00:00 2021-11-20T05:26:03.081 INFO:teuthology.orchestra.run.smithi187.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 36 MB/s | 486 kB 00:00 2021-11-20T05:26:03.180 INFO:teuthology.orchestra.run.smithi187.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2021-11-20T05:26:03.184 INFO:teuthology.orchestra.run.smithi187.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 9.9 MB/s | 44 kB 00:00 2021-11-20T05:26:03.188 INFO:teuthology.orchestra.run.smithi187.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 28 MB/s | 80 kB 00:00 2021-11-20T05:26:03.194 INFO:teuthology.orchestra.run.smithi187.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 32 MB/s | 213 kB 00:00 2021-11-20T05:26:03.205 INFO:teuthology.orchestra.run.smithi187.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 40 MB/s | 387 kB 00:00 2021-11-20T05:26:03.213 INFO:teuthology.orchestra.run.smithi187.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 41 MB/s | 368 kB 00:00 2021-11-20T05:26:03.217 INFO:teuthology.orchestra.run.smithi187.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 27 MB/s | 114 kB 00:00 2021-11-20T05:26:03.283 INFO:teuthology.orchestra.run.smithi187.stdout:(12/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 2.2 MB/s | 145 kB 00:00 2021-11-20T05:26:03.310 INFO:teuthology.orchestra.run.smithi187.stdout:(13/27): librdkafka-0.11.4-3.el8.x86_64.rpm 13 MB/s | 354 kB 00:00 2021-11-20T05:26:03.316 INFO:teuthology.orchestra.run.smithi187.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 42 MB/s | 259 kB 00:00 2021-11-20T05:26:03.318 INFO:teuthology.orchestra.run.smithi187.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 24 MB/s | 44 kB 00:00 2021-11-20T05:26:03.337 INFO:teuthology.orchestra.run.smithi187.stdout:(16/27): daxctl-libs-71.1-2.el8.x86_64.rpm 2.1 MB/s | 41 kB 00:00 2021-11-20T05:26:03.364 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:26:03.367 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:26:03.367 INFO:teuthology.orchestra.run.smithi058.stdout: Package Arch Version Repository Size 2021-11-20T05:26:03.367 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:26:03.368 INFO:teuthology.orchestra.run.smithi058.stdout:Installing: 2021-11-20T05:26:03.368 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2021-11-20T05:26:03.368 INFO:teuthology.orchestra.run.smithi058.stdout:Upgrading: 2021-11-20T05:26:03.368 INFO:teuthology.orchestra.run.smithi058.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-20T05:26:03.369 INFO:teuthology.orchestra.run.smithi058.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2021-11-20T05:26:03.369 INFO:teuthology.orchestra.run.smithi058.stdout:Installing dependencies: 2021-11-20T05:26:03.369 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2021-11-20T05:26:03.369 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2021-11-20T05:26:03.370 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2021-11-20T05:26:03.370 INFO:teuthology.orchestra.run.smithi058.stdout: daxctl-libs x86_64 71.1-2.el8 CentOS-Base 41 k 2021-11-20T05:26:03.370 INFO:teuthology.orchestra.run.smithi058.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-11-20T05:26:03.370 INFO:teuthology.orchestra.run.smithi058.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-11-20T05:26:03.371 INFO:teuthology.orchestra.run.smithi058.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-11-20T05:26:03.371 INFO:teuthology.orchestra.run.smithi058.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2021-11-20T05:26:03.371 INFO:teuthology.orchestra.run.smithi058.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-11-20T05:26:03.371 INFO:teuthology.orchestra.run.smithi058.stdout: libpmemobj x86_64 1.6.1-1.el8 CentOS-AppStream 145 k 2021-11-20T05:26:03.371 INFO:teuthology.orchestra.run.smithi058.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-11-20T05:26:03.372 INFO:teuthology.orchestra.run.smithi058.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2021-11-20T05:26:03.372 INFO:teuthology.orchestra.run.smithi058.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2021-11-20T05:26:03.372 INFO:teuthology.orchestra.run.smithi058.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2021-11-20T05:26:03.372 INFO:teuthology.orchestra.run.smithi058.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-11-20T05:26:03.373 INFO:teuthology.orchestra.run.smithi058.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-11-20T05:26:03.373 INFO:teuthology.orchestra.run.smithi058.stdout: ndctl-libs x86_64 71.1-2.el8 CentOS-Base 78 k 2021-11-20T05:26:03.373 INFO:teuthology.orchestra.run.smithi058.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2021-11-20T05:26:03.373 INFO:teuthology.orchestra.run.smithi058.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2021-11-20T05:26:03.373 INFO:teuthology.orchestra.run.smithi058.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2021-11-20T05:26:03.374 INFO:teuthology.orchestra.run.smithi058.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-11-20T05:26:03.374 INFO:teuthology.orchestra.run.smithi058.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2021-11-20T05:26:03.374 INFO:teuthology.orchestra.run.smithi058.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2021-11-20T05:26:03.374 INFO:teuthology.orchestra.run.smithi058.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2021-11-20T05:26:03.375 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:26:03.375 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:26:03.375 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:26:03.375 INFO:teuthology.orchestra.run.smithi058.stdout:Install 25 Packages 2021-11-20T05:26:03.376 INFO:teuthology.orchestra.run.smithi058.stdout:Upgrade 2 Packages 2021-11-20T05:26:03.376 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:26:03.376 INFO:teuthology.orchestra.run.smithi058.stdout:Total download size: 59 M 2021-11-20T05:26:03.377 INFO:teuthology.orchestra.run.smithi058.stdout:Downloading Packages: 2021-11-20T05:26:03.453 INFO:teuthology.orchestra.run.smithi187.stdout:(17/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 22 MB/s | 13 MB 00:00 2021-11-20T05:26:03.453 INFO:teuthology.orchestra.run.smithi187.stdout:(18/27): librabbitmq-0.9.0-3.el8.x86_64.rpm 404 kB/s | 47 kB 00:00 2021-11-20T05:26:03.558 INFO:teuthology.orchestra.run.smithi058.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 32 MB/s | 6.0 MB 00:00 2021-11-20T05:26:03.563 INFO:teuthology.orchestra.run.smithi058.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 4.7 MB/s | 25 kB 00:00 2021-11-20T05:26:03.586 INFO:teuthology.orchestra.run.smithi058.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 35 MB/s | 801 kB 00:00 2021-11-20T05:26:03.602 INFO:teuthology.orchestra.run.smithi058.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 30 MB/s | 486 kB 00:00 2021-11-20T05:26:03.688 INFO:teuthology.orchestra.run.smithi187.stdout:(19/27): ceph-common-16.2.4-0.el8.x86_64.rpm 30 MB/s | 24 MB 00:00 2021-11-20T05:26:03.689 INFO:teuthology.orchestra.run.smithi187.stdout:(20/27): ndctl-libs-71.1-2.el8.x86_64.rpm 333 kB/s | 78 kB 00:00 2021-11-20T05:26:03.689 INFO:teuthology.orchestra.run.smithi187.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 493 kB/s | 116 kB 00:00 2021-11-20T05:26:03.720 INFO:teuthology.orchestra.run.smithi058.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 35 MB/s | 4.1 MB 00:00 2021-11-20T05:26:03.720 INFO:teuthology.orchestra.run.smithi187.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 1.9 MB/s | 59 kB 00:00 2021-11-20T05:26:03.731 INFO:teuthology.orchestra.run.smithi058.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 3.7 MB/s | 44 kB 00:00 2021-11-20T05:26:03.743 INFO:teuthology.orchestra.run.smithi058.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 7.2 MB/s | 80 kB 00:00 2021-11-20T05:26:03.754 INFO:teuthology.orchestra.run.smithi187.stdout:(23/27): libunwind-1.3.1-3.el8.x86_64.rpm 2.2 MB/s | 75 kB 00:00 2021-11-20T05:26:03.756 INFO:teuthology.orchestra.run.smithi058.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 15 MB/s | 213 kB 00:00 2021-11-20T05:26:03.775 INFO:teuthology.orchestra.run.smithi058.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 20 MB/s | 387 kB 00:00 2021-11-20T05:26:03.796 INFO:teuthology.orchestra.run.smithi058.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 17 MB/s | 368 kB 00:00 2021-11-20T05:26:03.813 INFO:teuthology.orchestra.run.smithi058.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 6.8 MB/s | 114 kB 00:00 2021-11-20T05:26:03.828 INFO:teuthology.orchestra.run.smithi187.stdout:(24/27): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2021-11-20T05:26:03.855 INFO:teuthology.orchestra.run.smithi187.stdout:(25/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2021-11-20T05:26:03.885 INFO:teuthology.orchestra.run.smithi187.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 29 MB/s | 3.7 MB 00:00 2021-11-20T05:26:03.920 INFO:teuthology.orchestra.run.smithi187.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 44 MB/s | 4.0 MB 00:00 2021-11-20T05:26:03.925 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:26:03.925 INFO:teuthology.orchestra.run.smithi187.stdout:Total 56 MB/s | 59 MB 00:01 2021-11-20T05:26:03.936 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2021-11-20T05:26:03.937 INFO:teuthology.orchestra.run.smithi058.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 22 MB/s | 13 MB 00:00 2021-11-20T05:26:03.984 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2021-11-20T05:26:03.984 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2021-11-20T05:26:04.123 INFO:teuthology.orchestra.run.smithi058.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 32 MB/s | 24 MB 00:00 2021-11-20T05:26:04.188 INFO:teuthology.orchestra.run.smithi058.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 388 kB/s | 145 kB 00:00 2021-11-20T05:26:04.213 INFO:teuthology.orchestra.run.smithi058.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 2.1 MB/s | 44 kB 00:00 2021-11-20T05:26:04.225 INFO:teuthology.orchestra.run.smithi058.stdout:(16/27): daxctl-libs-71.1-2.el8.x86_64.rpm 3.2 MB/s | 41 kB 00:00 2021-11-20T05:26:04.230 INFO:teuthology.orchestra.run.smithi058.stdout:(17/27): librabbitmq-0.9.0-3.el8.x86_64.rpm 10 MB/s | 47 kB 00:00 2021-11-20T05:26:04.235 INFO:teuthology.orchestra.run.smithi058.stdout:(18/27): librdkafka-0.11.4-3.el8.x86_64.rpm 1.2 MB/s | 354 kB 00:00 2021-11-20T05:26:04.236 INFO:teuthology.orchestra.run.smithi058.stdout:(19/27): ndctl-libs-71.1-2.el8.x86_64.rpm 14 MB/s | 78 kB 00:00 2021-11-20T05:26:04.237 INFO:teuthology.orchestra.run.smithi058.stdout:(20/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.2 MB/s | 259 kB 00:00 2021-11-20T05:26:04.318 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2021-11-20T05:26:04.321 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2021-11-20T05:26:04.374 INFO:teuthology.orchestra.run.smithi058.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 840 kB/s | 116 kB 00:00 2021-11-20T05:26:04.376 INFO:teuthology.orchestra.run.smithi058.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2021-11-20T05:26:04.398 INFO:teuthology.orchestra.run.smithi058.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2021-11-20T05:26:04.402 INFO:teuthology.orchestra.run.smithi058.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 2.0 MB/s | 59 kB 00:00 2021-11-20T05:26:04.405 INFO:teuthology.orchestra.run.smithi058.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 2.6 MB/s | 75 kB 00:00 2021-11-20T05:26:04.478 INFO:teuthology.orchestra.run.smithi058.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 54 MB/s | 4.0 MB 00:00 2021-11-20T05:26:04.491 INFO:teuthology.orchestra.run.smithi058.stdout:(27/27): librados2-16.2.4-0.el8.x86_64.rpm 40 MB/s | 3.7 MB 00:00 2021-11-20T05:26:04.495 INFO:teuthology.orchestra.run.smithi058.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:26:04.496 INFO:teuthology.orchestra.run.smithi058.stdout:Total 52 MB/s | 59 MB 00:01 2021-11-20T05:26:04.508 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction check 2021-11-20T05:26:04.560 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction check succeeded. 2021-11-20T05:26:04.561 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction test 2021-11-20T05:26:04.918 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction test succeeded. 2021-11-20T05:26:04.922 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction 2021-11-20T05:26:05.497 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2021-11-20T05:26:05.670 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2021-11-20T05:26:05.858 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2021-11-20T05:26:05.997 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-20T05:26:06.130 INFO:teuthology.orchestra.run.smithi058.stdout: Preparing : 1/1 2021-11-20T05:26:06.273 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2021-11-20T05:26:06.429 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-20T05:26:06.454 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2021-11-20T05:26:06.614 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T05:26:06.630 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-20T05:26:06.711 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T05:26:06.946 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2021-11-20T05:26:07.074 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-20T05:26:07.089 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T05:26:07.173 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T05:26:07.266 INFO:teuthology.orchestra.run.smithi058.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T05:26:07.362 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/29 2021-11-20T05:26:07.367 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T05:26:07.530 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-20T05:26:07.561 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2021-11-20T05:26:07.724 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T05:26:07.794 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T05:26:07.936 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-20T05:26:07.975 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/29 2021-11-20T05:26:08.063 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T05:26:08.149 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-20T05:26:08.171 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T05:26:08.359 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2021-11-20T05:26:08.513 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-20T05:26:08.579 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-20T05:26:08.599 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-20T05:26:08.724 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T05:26:08.752 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2021-11-20T05:26:08.889 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2021-11-20T05:26:08.893 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T05:26:09.151 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-20T05:26:09.473 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2021-11-20T05:26:09.573 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-20T05:26:09.683 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-20T05:26:09.730 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-20T05:26:09.777 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-20T05:26:09.916 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2021-11-20T05:26:10.085 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2021-11-20T05:26:10.178 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-20T05:26:10.215 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-20T05:26:10.319 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-20T05:26:10.364 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-20T05:26:10.461 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-20T05:26:10.465 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-20T05:26:10.658 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2021-11-20T05:26:10.881 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2021-11-20T05:26:10.947 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-20T05:26:11.022 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-20T05:26:11.124 INFO:teuthology.orchestra.run.smithi058.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-20T05:26:11.129 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-20T05:26:11.209 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-20T05:26:11.305 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2021-11-20T05:26:11.387 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2021-11-20T05:26:11.497 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2021-11-20T05:26:11.603 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2021-11-20T05:26:11.725 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2021-11-20T05:26:11.765 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-20T05:26:11.833 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-20T05:26:11.947 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2021-11-20T05:26:12.034 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2021-11-20T05:26:12.241 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2021-11-20T05:26:12.476 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2021-11-20T05:26:12.707 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2021-11-20T05:26:16.208 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T05:26:16.429 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T05:26:17.125 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T05:26:17.448 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-20T05:26:17.548 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-20T05:26:17.695 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T05:26:17.799 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T05:26:18.007 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T05:26:18.745 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T05:26:19.052 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-20T05:26:19.190 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-20T05:26:19.322 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T05:26:36.212 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T05:26:36.213 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /sys 2021-11-20T05:26:36.213 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /proc 2021-11-20T05:26:36.213 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /dev 2021-11-20T05:26:36.214 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /run 2021-11-20T05:26:36.214 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /mnt 2021-11-20T05:26:36.214 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /var/tmp 2021-11-20T05:26:36.214 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /home 2021-11-20T05:26:36.214 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /tmp 2021-11-20T05:26:36.215 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /dev 2021-11-20T05:26:36.215 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:26:37.519 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-20T05:26:37.595 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-20T05:26:37.595 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-20T05:26:37.810 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-20T05:26:37.811 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T05:26:37.870 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T05:26:37.870 INFO:teuthology.orchestra.run.smithi058.stdout:skipping the directory /sys 2021-11-20T05:26:37.871 INFO:teuthology.orchestra.run.smithi058.stdout:skipping the directory /proc 2021-11-20T05:26:37.871 INFO:teuthology.orchestra.run.smithi058.stdout:skipping the directory /dev 2021-11-20T05:26:37.871 INFO:teuthology.orchestra.run.smithi058.stdout:skipping the directory /run 2021-11-20T05:26:37.871 INFO:teuthology.orchestra.run.smithi058.stdout:skipping the directory /mnt 2021-11-20T05:26:37.872 INFO:teuthology.orchestra.run.smithi058.stdout:skipping the directory /var/tmp 2021-11-20T05:26:37.872 INFO:teuthology.orchestra.run.smithi058.stdout:skipping the directory /home 2021-11-20T05:26:37.872 INFO:teuthology.orchestra.run.smithi058.stdout:skipping the directory /tmp 2021-11-20T05:26:37.872 INFO:teuthology.orchestra.run.smithi058.stdout:skipping the directory /dev 2021-11-20T05:26:37.872 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:26:38.548 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T05:26:38.548 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2021-11-20T05:26:38.549 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2021-11-20T05:26:38.549 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2021-11-20T05:26:38.549 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T05:26:38.549 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2021-11-20T05:26:38.550 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T05:26:38.550 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2021-11-20T05:26:38.550 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2021-11-20T05:26:38.550 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T05:26:38.551 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2021-11-20T05:26:38.551 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2021-11-20T05:26:38.551 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2021-11-20T05:26:38.551 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2021-11-20T05:26:38.552 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2021-11-20T05:26:38.552 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2021-11-20T05:26:38.552 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2021-11-20T05:26:38.552 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2021-11-20T05:26:38.552 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : daxctl-libs-71.1-2.el8.x86_64 18/29 2021-11-20T05:26:38.553 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 19/29 2021-11-20T05:26:38.554 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ndctl-libs-71.1-2.el8.x86_64 20/29 2021-11-20T05:26:38.555 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2021-11-20T05:26:38.555 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2021-11-20T05:26:38.555 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2021-11-20T05:26:38.555 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2021-11-20T05:26:38.556 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2021-11-20T05:26:38.556 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T05:26:38.556 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2021-11-20T05:26:38.556 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2021-11-20T05:26:39.382 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-20T05:26:39.470 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T05:26:39.470 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:26:39.471 INFO:teuthology.orchestra.run.smithi187.stdout:Upgraded: 2021-11-20T05:26:39.471 INFO:teuthology.orchestra.run.smithi187.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:39.471 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2021-11-20T05:26:39.471 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:39.472 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:39.472 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:39.472 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:39.472 INFO:teuthology.orchestra.run.smithi187.stdout: daxctl-libs-71.1-2.el8.x86_64 2021-11-20T05:26:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: fmt-6.2.1-1.el8.x86_64 2021-11-20T05:26:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-11-20T05:26:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: leveldb-1.22-1.el8.x86_64 2021-11-20T05:26:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: liboath-2.6.2-3.el8.x86_64 2021-11-20T05:26:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: libpmemobj-1.6.1-1.el8.x86_64 2021-11-20T05:26:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-11-20T05:26:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka-0.11.4-3.el8.x86_64 2021-11-20T05:26:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind-1.3.1-3.el8.x86_64 2021-11-20T05:26:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-11-20T05:26:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: ndctl-libs-71.1-2.el8.x86_64 2021-11-20T05:26:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-11-20T05:26:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:39.477 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:39.477 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:39.477 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:26:39.477 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:26:39.478 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-20T05:26:39.478 INFO:teuthology.orchestra.run.smithi058.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-20T05:26:39.696 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-20T05:26:39.697 INFO:teuthology.orchestra.run.smithi058.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T05:26:39.747 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-test 2021-11-20T05:26:40.302 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:40 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:26:40.504 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T05:26:40.505 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2021-11-20T05:26:40.505 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2021-11-20T05:26:40.505 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2021-11-20T05:26:40.506 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T05:26:40.506 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2021-11-20T05:26:40.506 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T05:26:40.506 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2021-11-20T05:26:40.506 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2021-11-20T05:26:40.507 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T05:26:40.507 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2021-11-20T05:26:40.507 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2021-11-20T05:26:40.508 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2021-11-20T05:26:40.508 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2021-11-20T05:26:40.508 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2021-11-20T05:26:40.508 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2021-11-20T05:26:40.509 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2021-11-20T05:26:40.509 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2021-11-20T05:26:40.509 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : daxctl-libs-71.1-2.el8.x86_64 18/29 2021-11-20T05:26:40.509 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 19/29 2021-11-20T05:26:40.509 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ndctl-libs-71.1-2.el8.x86_64 20/29 2021-11-20T05:26:40.510 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2021-11-20T05:26:40.510 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2021-11-20T05:26:40.510 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2021-11-20T05:26:40.510 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2021-11-20T05:26:40.510 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2021-11-20T05:26:40.511 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T05:26:40.511 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2021-11-20T05:26:40.511 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2021-11-20T05:26:41.418 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T05:26:41.419 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:26:41.419 INFO:teuthology.orchestra.run.smithi058.stdout:Upgraded: 2021-11-20T05:26:41.419 INFO:teuthology.orchestra.run.smithi058.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:41.419 INFO:teuthology.orchestra.run.smithi058.stdout:Installed: 2021-11-20T05:26:41.420 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:41.420 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:41.420 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:41.420 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:41.421 INFO:teuthology.orchestra.run.smithi058.stdout: daxctl-libs-71.1-2.el8.x86_64 2021-11-20T05:26:41.421 INFO:teuthology.orchestra.run.smithi058.stdout: fmt-6.2.1-1.el8.x86_64 2021-11-20T05:26:41.421 INFO:teuthology.orchestra.run.smithi058.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-11-20T05:26:41.421 INFO:teuthology.orchestra.run.smithi058.stdout: leveldb-1.22-1.el8.x86_64 2021-11-20T05:26:41.421 INFO:teuthology.orchestra.run.smithi058.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:41.422 INFO:teuthology.orchestra.run.smithi058.stdout: liboath-2.6.2-3.el8.x86_64 2021-11-20T05:26:41.422 INFO:teuthology.orchestra.run.smithi058.stdout: libpmemobj-1.6.1-1.el8.x86_64 2021-11-20T05:26:41.422 INFO:teuthology.orchestra.run.smithi058.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-11-20T05:26:41.422 INFO:teuthology.orchestra.run.smithi058.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:41.423 INFO:teuthology.orchestra.run.smithi058.stdout: librdkafka-0.11.4-3.el8.x86_64 2021-11-20T05:26:41.423 INFO:teuthology.orchestra.run.smithi058.stdout: librgw2-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:41.423 INFO:teuthology.orchestra.run.smithi058.stdout: libunwind-1.3.1-3.el8.x86_64 2021-11-20T05:26:41.423 INFO:teuthology.orchestra.run.smithi058.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-11-20T05:26:41.424 INFO:teuthology.orchestra.run.smithi058.stdout: ndctl-libs-71.1-2.el8.x86_64 2021-11-20T05:26:41.424 INFO:teuthology.orchestra.run.smithi058.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:41.424 INFO:teuthology.orchestra.run.smithi058.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:41.424 INFO:teuthology.orchestra.run.smithi058.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:41.424 INFO:teuthology.orchestra.run.smithi058.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-11-20T05:26:41.425 INFO:teuthology.orchestra.run.smithi058.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:41.425 INFO:teuthology.orchestra.run.smithi058.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:41.425 INFO:teuthology.orchestra.run.smithi058.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2021-11-20T05:26:41.425 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:26:41.426 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:26:41.598 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:26:41.599 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:26:41.599 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2021-11-20T05:26:41.599 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:26:41.599 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2021-11-20T05:26:41.600 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2021-11-20T05:26:41.600 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2021-11-20T05:26:41.600 INFO:teuthology.orchestra.run.smithi187.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2021-11-20T05:26:41.600 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2021-11-20T05:26:41.600 INFO:teuthology.orchestra.run.smithi187.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-11-20T05:26:41.601 INFO:teuthology.orchestra.run.smithi187.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2021-11-20T05:26:41.601 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-11-20T05:26:41.601 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:26:41.601 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:26:41.602 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:26:41.602 INFO:teuthology.orchestra.run.smithi187.stdout:Install 6 Packages 2021-11-20T05:26:41.602 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:26:41.603 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 53 M 2021-11-20T05:26:41.603 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 229 M 2021-11-20T05:26:41.603 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2021-11-20T05:26:41.632 INFO:teuthology.orchestra.run.smithi187.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 5.5 MB/s | 169 kB 00:00 2021-11-20T05:26:41.645 INFO:teuthology.orchestra.run.smithi187.stdout:(2/6): jq-1.6-3.el8.x86_64.rpm 4.6 MB/s | 202 kB 00:00 2021-11-20T05:26:41.649 INFO:teuthology.orchestra.run.smithi187.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 67 MB/s | 323 kB 00:00 2021-11-20T05:26:41.664 INFO:teuthology.orchestra.run.smithi187.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 5.8 MB/s | 187 kB 00:00 2021-11-20T05:26:41.692 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install ceph-test 2021-11-20T05:26:41.763 INFO:teuthology.orchestra.run.smithi187.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 609 kB/s | 69 kB 00:00 2021-11-20T05:26:42.276 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:00:42 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:26:42.545 INFO:teuthology.orchestra.run.smithi187.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 55 MB/s | 52 MB 00:00 2021-11-20T05:26:42.550 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:26:42.551 INFO:teuthology.orchestra.run.smithi187.stdout:Total 56 MB/s | 53 MB 00:00 2021-11-20T05:26:42.557 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2021-11-20T05:26:42.581 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2021-11-20T05:26:42.581 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2021-11-20T05:26:42.869 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2021-11-20T05:26:42.872 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2021-11-20T05:26:43.417 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2021-11-20T05:26:43.586 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2021-11-20T05:26:43.650 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:26:43.651 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:26:43.651 INFO:teuthology.orchestra.run.smithi058.stdout: Package Arch Version Repository Size 2021-11-20T05:26:43.651 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:26:43.652 INFO:teuthology.orchestra.run.smithi058.stdout:Installing: 2021-11-20T05:26:43.652 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2021-11-20T05:26:43.652 INFO:teuthology.orchestra.run.smithi058.stdout:Installing dependencies: 2021-11-20T05:26:43.652 INFO:teuthology.orchestra.run.smithi058.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2021-11-20T05:26:43.653 INFO:teuthology.orchestra.run.smithi058.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2021-11-20T05:26:43.653 INFO:teuthology.orchestra.run.smithi058.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-11-20T05:26:43.653 INFO:teuthology.orchestra.run.smithi058.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2021-11-20T05:26:43.653 INFO:teuthology.orchestra.run.smithi058.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-11-20T05:26:43.653 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:26:43.654 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:26:43.654 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:26:43.654 INFO:teuthology.orchestra.run.smithi058.stdout:Install 6 Packages 2021-11-20T05:26:43.654 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:26:43.655 INFO:teuthology.orchestra.run.smithi058.stdout:Total download size: 53 M 2021-11-20T05:26:43.655 INFO:teuthology.orchestra.run.smithi058.stdout:Installed size: 229 M 2021-11-20T05:26:43.656 INFO:teuthology.orchestra.run.smithi058.stdout:Downloading Packages: 2021-11-20T05:26:43.679 INFO:teuthology.orchestra.run.smithi058.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 6.5 MB/s | 169 kB 00:00 2021-11-20T05:26:43.698 INFO:teuthology.orchestra.run.smithi058.stdout:(2/6): jq-1.6-3.el8.x86_64.rpm 4.5 MB/s | 202 kB 00:00 2021-11-20T05:26:43.704 INFO:teuthology.orchestra.run.smithi058.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 62 MB/s | 323 kB 00:00 2021-11-20T05:26:43.711 INFO:teuthology.orchestra.run.smithi058.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 5.9 MB/s | 187 kB 00:00 2021-11-20T05:26:43.785 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2021-11-20T05:26:43.815 INFO:teuthology.orchestra.run.smithi058.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 621 kB/s | 69 kB 00:00 2021-11-20T05:26:43.973 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-20T05:26:44.052 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-20T05:26:44.231 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2021-11-20T05:26:44.423 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-20T05:26:44.591 INFO:teuthology.orchestra.run.smithi058.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 55 MB/s | 52 MB 00:00 2021-11-20T05:26:44.595 INFO:teuthology.orchestra.run.smithi058.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:26:44.596 INFO:teuthology.orchestra.run.smithi058.stdout:Total 56 MB/s | 53 MB 00:00 2021-11-20T05:26:44.603 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction check 2021-11-20T05:26:44.630 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction check succeeded. 2021-11-20T05:26:44.631 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction test 2021-11-20T05:26:44.934 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction test succeeded. 2021-11-20T05:26:44.938 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction 2021-11-20T05:26:45.465 INFO:teuthology.orchestra.run.smithi058.stdout: Preparing : 1/1 2021-11-20T05:26:45.625 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2021-11-20T05:26:45.823 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2021-11-20T05:26:46.026 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-20T05:26:46.096 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-20T05:26:46.268 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2021-11-20T05:26:46.468 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-20T05:26:49.213 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-20T05:26:49.471 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-20T05:26:49.725 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-20T05:26:49.725 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2021-11-20T05:26:49.726 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2021-11-20T05:26:49.726 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2021-11-20T05:26:49.726 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-11-20T05:26:49.726 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2021-11-20T05:26:50.144 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-11-20T05:26:50.144 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:26:50.144 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2021-11-20T05:26:50.145 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-3.el8.x86_64 2021-11-20T05:26:50.145 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-11-20T05:26:50.145 INFO:teuthology.orchestra.run.smithi187.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-11-20T05:26:50.145 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:26:50.146 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:26:50.339 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph 2021-11-20T05:26:50.876 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:50 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:26:51.568 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-20T05:26:51.808 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-20T05:26:52.062 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-20T05:26:52.062 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2021-11-20T05:26:52.062 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2021-11-20T05:26:52.063 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2021-11-20T05:26:52.063 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-11-20T05:26:52.063 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2021-11-20T05:26:52.193 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:26:52.196 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:26:52.196 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2021-11-20T05:26:52.196 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:26:52.196 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2021-11-20T05:26:52.197 INFO:teuthology.orchestra.run.smithi187.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2021-11-20T05:26:52.197 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2021-11-20T05:26:52.197 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2021-11-20T05:26:52.197 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2021-11-20T05:26:52.198 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2021-11-20T05:26:52.198 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2021-11-20T05:26:52.198 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2021-11-20T05:26:52.198 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-11-20T05:26:52.199 INFO:teuthology.orchestra.run.smithi187.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-11-20T05:26:52.199 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-11-20T05:26:52.199 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-11-20T05:26:52.199 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-11-20T05:26:52.200 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2021-11-20T05:26:52.200 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-11-20T05:26:52.200 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-11-20T05:26:52.200 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-11-20T05:26:52.200 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-11-20T05:26:52.201 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-11-20T05:26:52.201 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-11-20T05:26:52.201 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-11-20T05:26:52.201 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-11-20T05:26:52.202 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-11-20T05:26:52.202 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-11-20T05:26:52.202 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-11-20T05:26:52.202 INFO:teuthology.orchestra.run.smithi187.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-11-20T05:26:52.202 INFO:teuthology.orchestra.run.smithi187.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-11-20T05:26:52.203 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-11-20T05:26:52.203 INFO:teuthology.orchestra.run.smithi187.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-11-20T05:26:52.203 INFO:teuthology.orchestra.run.smithi187.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-11-20T05:26:52.203 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-11-20T05:26:52.204 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-11-20T05:26:52.204 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-11-20T05:26:52.204 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-11-20T05:26:52.204 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:26:52.204 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:26:52.205 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:26:52.205 INFO:teuthology.orchestra.run.smithi187.stdout:Install 32 Packages 2021-11-20T05:26:52.205 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:26:52.206 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 31 M 2021-11-20T05:26:52.206 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 111 M 2021-11-20T05:26:52.206 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2021-11-20T05:26:52.220 INFO:teuthology.orchestra.run.smithi187.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 333 kB/s | 6.3 kB 00:00 2021-11-20T05:26:52.262 INFO:teuthology.orchestra.run.smithi187.stdout:(2/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 40 MB/s | 2.4 MB 00:00 2021-11-20T05:26:52.308 INFO:teuthology.orchestra.run.smithi187.stdout:(3/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2021-11-20T05:26:52.323 INFO:teuthology.orchestra.run.smithi187.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 15 MB/s | 220 kB 00:00 2021-11-20T05:26:52.355 INFO:teuthology.orchestra.run.smithi187.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 4.8 MB/s | 157 kB 00:00 2021-11-20T05:26:52.357 INFO:teuthology.orchestra.run.smithi187.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 29 MB/s | 39 kB 00:00 2021-11-20T05:26:52.359 INFO:teuthology.orchestra.run.smithi187.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 42 MB/s | 103 kB 00:00 2021-11-20T05:26:52.361 INFO:teuthology.orchestra.run.smithi187.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 33 MB/s | 54 kB 00:00 2021-11-20T05:26:52.368 INFO:teuthology.orchestra.run.smithi187.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 71 MB/s | 457 kB 00:00 2021-11-20T05:26:52.372 INFO:teuthology.orchestra.run.smithi187.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 63 MB/s | 237 kB 00:00 2021-11-20T05:26:52.379 INFO:teuthology.orchestra.run.smithi187.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 78 MB/s | 559 kB 00:00 2021-11-20T05:26:52.384 INFO:teuthology.orchestra.run.smithi187.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 22 MB/s | 109 kB 00:00 2021-11-20T05:26:52.410 INFO:teuthology.orchestra.run.smithi187.stdout:(13/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 26 MB/s | 5.0 MB 00:00 2021-11-20T05:26:52.512 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-11-20T05:26:52.512 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:26:52.512 INFO:teuthology.orchestra.run.smithi058.stdout:Installed: 2021-11-20T05:26:52.513 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-3.el8.x86_64 2021-11-20T05:26:52.513 INFO:teuthology.orchestra.run.smithi058.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-11-20T05:26:52.513 INFO:teuthology.orchestra.run.smithi058.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-11-20T05:26:52.513 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:26:52.514 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:26:52.597 INFO:teuthology.orchestra.run.smithi187.stdout:(14/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 54 MB/s | 18 MB 00:00 2021-11-20T05:26:52.598 INFO:teuthology.orchestra.run.smithi187.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 204 kB/s | 44 kB 00:00 2021-11-20T05:26:52.599 INFO:teuthology.orchestra.run.smithi187.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 980 kB/s | 185 kB 00:00 2021-11-20T05:26:52.627 INFO:teuthology.orchestra.run.smithi187.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 401 kB/s | 11 kB 00:00 2021-11-20T05:26:52.656 INFO:teuthology.orchestra.run.smithi187.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 631 kB/s | 18 kB 00:00 2021-11-20T05:26:52.671 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install ceph 2021-11-20T05:26:52.682 INFO:teuthology.orchestra.run.smithi187.stdout:(19/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.0 MB/s | 173 kB 00:00 2021-11-20T05:26:52.683 INFO:teuthology.orchestra.run.smithi187.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 1.7 MB/s | 49 kB 00:00 2021-11-20T05:26:52.712 INFO:teuthology.orchestra.run.smithi187.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 2.0 MB/s | 59 kB 00:00 2021-11-20T05:26:52.741 INFO:teuthology.orchestra.run.smithi187.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 4.8 MB/s | 283 kB 00:00 2021-11-20T05:26:52.741 INFO:teuthology.orchestra.run.smithi187.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 532 kB/s | 16 kB 00:00 2021-11-20T05:26:52.769 INFO:teuthology.orchestra.run.smithi187.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 663 kB/s | 19 kB 00:00 2021-11-20T05:26:52.771 INFO:teuthology.orchestra.run.smithi187.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 836 kB/s | 24 kB 00:00 2021-11-20T05:26:52.797 INFO:teuthology.orchestra.run.smithi187.stdout:(26/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.9 MB/s | 384 kB 00:00 2021-11-20T05:26:52.798 INFO:teuthology.orchestra.run.smithi187.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 1.0 MB/s | 29 kB 00:00 2021-11-20T05:26:52.799 INFO:teuthology.orchestra.run.smithi187.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 954 kB/s | 27 kB 00:00 2021-11-20T05:26:52.829 INFO:teuthology.orchestra.run.smithi187.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 7.7 MB/s | 251 kB 00:00 2021-11-20T05:26:52.830 INFO:teuthology.orchestra.run.smithi187.stdout:(30/32): python3-waitress-1.2.1-2.el8.1.noarch. 7.0 MB/s | 241 kB 00:00 2021-11-20T05:26:52.831 INFO:teuthology.orchestra.run.smithi187.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 2.6 MB/s | 85 kB 00:00 2021-11-20T05:26:52.858 INFO:teuthology.orchestra.run.smithi187.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 797 kB/s | 23 kB 00:00 2021-11-20T05:26:52.861 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:26:52.862 INFO:teuthology.orchestra.run.smithi187.stdout:Total 47 MB/s | 31 MB 00:00 2021-11-20T05:26:52.875 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2021-11-20T05:26:52.914 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2021-11-20T05:26:52.914 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2021-11-20T05:26:53.133 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2021-11-20T05:26:53.136 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2021-11-20T05:26:53.234 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:00:53 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:26:54.330 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2021-11-20T05:26:54.572 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-11-20T05:26:54.595 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:26:54.598 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:26:54.599 INFO:teuthology.orchestra.run.smithi058.stdout: Package Arch Version Repository Size 2021-11-20T05:26:54.599 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:26:54.599 INFO:teuthology.orchestra.run.smithi058.stdout:Installing: 2021-11-20T05:26:54.599 INFO:teuthology.orchestra.run.smithi058.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2021-11-20T05:26:54.599 INFO:teuthology.orchestra.run.smithi058.stdout:Installing dependencies: 2021-11-20T05:26:54.600 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2021-11-20T05:26:54.600 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2021-11-20T05:26:54.600 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2021-11-20T05:26:54.600 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2021-11-20T05:26:54.601 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2021-11-20T05:26:54.601 INFO:teuthology.orchestra.run.smithi058.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-11-20T05:26:54.601 INFO:teuthology.orchestra.run.smithi058.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-11-20T05:26:54.601 INFO:teuthology.orchestra.run.smithi058.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-11-20T05:26:54.602 INFO:teuthology.orchestra.run.smithi058.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-11-20T05:26:54.602 INFO:teuthology.orchestra.run.smithi058.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-11-20T05:26:54.602 INFO:teuthology.orchestra.run.smithi058.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2021-11-20T05:26:54.602 INFO:teuthology.orchestra.run.smithi058.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-11-20T05:26:54.602 INFO:teuthology.orchestra.run.smithi058.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-11-20T05:26:54.603 INFO:teuthology.orchestra.run.smithi058.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-11-20T05:26:54.603 INFO:teuthology.orchestra.run.smithi058.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-11-20T05:26:54.603 INFO:teuthology.orchestra.run.smithi058.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-11-20T05:26:54.604 INFO:teuthology.orchestra.run.smithi058.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-11-20T05:26:54.604 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-11-20T05:26:54.605 INFO:teuthology.orchestra.run.smithi058.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-11-20T05:26:54.605 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-11-20T05:26:54.605 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-11-20T05:26:54.606 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-11-20T05:26:54.606 INFO:teuthology.orchestra.run.smithi058.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-11-20T05:26:54.607 INFO:teuthology.orchestra.run.smithi058.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-11-20T05:26:54.607 INFO:teuthology.orchestra.run.smithi058.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-11-20T05:26:54.607 INFO:teuthology.orchestra.run.smithi058.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-11-20T05:26:54.608 INFO:teuthology.orchestra.run.smithi058.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-11-20T05:26:54.608 INFO:teuthology.orchestra.run.smithi058.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-11-20T05:26:54.609 INFO:teuthology.orchestra.run.smithi058.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-11-20T05:26:54.609 INFO:teuthology.orchestra.run.smithi058.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-11-20T05:26:54.609 INFO:teuthology.orchestra.run.smithi058.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-11-20T05:26:54.609 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:26:54.610 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:26:54.610 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:26:54.610 INFO:teuthology.orchestra.run.smithi058.stdout:Install 32 Packages 2021-11-20T05:26:54.610 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:26:54.611 INFO:teuthology.orchestra.run.smithi058.stdout:Total download size: 31 M 2021-11-20T05:26:54.611 INFO:teuthology.orchestra.run.smithi058.stdout:Installed size: 111 M 2021-11-20T05:26:54.612 INFO:teuthology.orchestra.run.smithi058.stdout:Downloading Packages: 2021-11-20T05:26:54.629 INFO:teuthology.orchestra.run.smithi058.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 256 kB/s | 6.3 kB 00:00 2021-11-20T05:26:54.668 INFO:teuthology.orchestra.run.smithi058.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 25 MB/s | 1.6 MB 00:00 2021-11-20T05:26:54.677 INFO:teuthology.orchestra.run.smithi058.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 33 MB/s | 2.4 MB 00:00 2021-11-20T05:26:54.683 INFO:teuthology.orchestra.run.smithi058.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 38 MB/s | 220 kB 00:00 2021-11-20T05:26:54.716 INFO:teuthology.orchestra.run.smithi058.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 4.7 MB/s | 157 kB 00:00 2021-11-20T05:26:54.717 INFO:teuthology.orchestra.run.smithi058.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 29 MB/s | 39 kB 00:00 2021-11-20T05:26:54.720 INFO:teuthology.orchestra.run.smithi058.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 39 MB/s | 103 kB 00:00 2021-11-20T05:26:54.722 INFO:teuthology.orchestra.run.smithi058.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 37 MB/s | 54 kB 00:00 2021-11-20T05:26:54.729 INFO:teuthology.orchestra.run.smithi058.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 67 MB/s | 457 kB 00:00 2021-11-20T05:26:54.733 INFO:teuthology.orchestra.run.smithi058.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 64 MB/s | 237 kB 00:00 2021-11-20T05:26:54.740 INFO:teuthology.orchestra.run.smithi058.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 73 MB/s | 559 kB 00:00 2021-11-20T05:26:54.743 INFO:teuthology.orchestra.run.smithi058.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 50 MB/s | 109 kB 00:00 2021-11-20T05:26:54.791 INFO:teuthology.orchestra.run.smithi058.stdout:(13/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 31 MB/s | 5.0 MB 00:00 2021-11-20T05:26:54.797 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-11-20T05:26:54.854 INFO:teuthology.orchestra.run.smithi058.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 394 kB/s | 44 kB 00:00 2021-11-20T05:26:55.017 INFO:teuthology.orchestra.run.smithi058.stdout:(15/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 52 MB/s | 18 MB 00:00 2021-11-20T05:26:55.018 INFO:teuthology.orchestra.run.smithi058.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.0 MB/s | 173 kB 00:00 2021-11-20T05:26:55.019 INFO:teuthology.orchestra.run.smithi058.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 811 kB/s | 185 kB 00:00 2021-11-20T05:26:55.037 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2021-11-20T05:26:55.047 INFO:teuthology.orchestra.run.smithi058.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 396 kB/s | 11 kB 00:00 2021-11-20T05:26:55.047 INFO:teuthology.orchestra.run.smithi058.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 631 kB/s | 18 kB 00:00 2021-11-20T05:26:55.075 INFO:teuthology.orchestra.run.smithi058.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 1.7 MB/s | 49 kB 00:00 2021-11-20T05:26:55.076 INFO:teuthology.orchestra.run.smithi058.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 2.0 MB/s | 59 kB 00:00 2021-11-20T05:26:55.105 INFO:teuthology.orchestra.run.smithi058.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 549 kB/s | 16 kB 00:00 2021-11-20T05:26:55.132 INFO:teuthology.orchestra.run.smithi058.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 4.9 MB/s | 283 kB 00:00 2021-11-20T05:26:55.134 INFO:teuthology.orchestra.run.smithi058.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 672 kB/s | 19 kB 00:00 2021-11-20T05:26:55.160 INFO:teuthology.orchestra.run.smithi058.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 853 kB/s | 24 kB 00:00 2021-11-20T05:26:55.161 INFO:teuthology.orchestra.run.smithi058.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 1.0 MB/s | 29 kB 00:00 2021-11-20T05:26:55.189 INFO:teuthology.orchestra.run.smithi058.stdout:(27/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.2 MB/s | 384 kB 00:00 2021-11-20T05:26:55.189 INFO:teuthology.orchestra.run.smithi058.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 950 kB/s | 27 kB 00:00 2021-11-20T05:26:55.217 INFO:teuthology.orchestra.run.smithi058.stdout:(29/32): python3-waitress-1.2.1-2.el8.1.noarch. 4.2 MB/s | 241 kB 00:00 2021-11-20T05:26:55.219 INFO:teuthology.orchestra.run.smithi058.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 2.9 MB/s | 85 kB 00:00 2021-11-20T05:26:55.221 INFO:teuthology.orchestra.run.smithi058.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 7.7 MB/s | 251 kB 00:00 2021-11-20T05:26:55.228 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2021-11-20T05:26:55.246 INFO:teuthology.orchestra.run.smithi058.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 791 kB/s | 23 kB 00:00 2021-11-20T05:26:55.250 INFO:teuthology.orchestra.run.smithi058.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:26:55.250 INFO:teuthology.orchestra.run.smithi058.stdout:Total 48 MB/s | 31 MB 00:00 2021-11-20T05:26:55.266 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction check 2021-11-20T05:26:55.308 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction check succeeded. 2021-11-20T05:26:55.309 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction test 2021-11-20T05:26:55.401 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2021-11-20T05:26:55.542 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction test succeeded. 2021-11-20T05:26:55.546 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction 2021-11-20T05:26:55.608 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2021-11-20T05:26:55.783 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2021-11-20T05:26:55.978 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2021-11-20T05:26:56.224 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2021-11-20T05:26:56.488 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2021-11-20T05:26:56.742 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2021-11-20T05:26:56.755 INFO:teuthology.orchestra.run.smithi058.stdout: Preparing : 1/1 2021-11-20T05:26:56.996 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-11-20T05:26:57.008 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2021-11-20T05:26:57.217 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-11-20T05:26:57.460 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2021-11-20T05:26:57.492 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2021-11-20T05:26:57.707 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2021-11-20T05:26:57.767 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2021-11-20T05:26:57.879 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2021-11-20T05:26:57.947 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2021-11-20T05:26:58.104 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2021-11-20T05:26:58.220 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2021-11-20T05:26:58.328 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2021-11-20T05:26:58.486 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-20T05:26:58.490 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2021-11-20T05:26:58.739 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2021-11-20T05:26:58.793 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2021-11-20T05:26:58.966 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2021-11-20T05:26:59.054 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2021-11-20T05:26:59.208 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2021-11-20T05:26:59.212 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2021-11-20T05:26:59.436 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-11-20T05:26:59.471 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2021-11-20T05:26:59.719 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-11-20T05:26:59.907 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2021-11-20T05:26:59.968 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-11-20T05:27:00.197 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2021-11-20T05:27:00.252 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2021-11-20T05:27:00.408 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2021-11-20T05:27:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2021-11-20T05:27:00.674 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2021-11-20T05:27:00.910 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-20T05:27:00.998 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-11-20T05:27:01.240 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2021-11-20T05:27:01.345 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2021-11-20T05:27:01.466 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2021-11-20T05:27:01.548 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-20T05:27:01.662 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2021-11-20T05:27:01.863 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-11-20T05:27:02.139 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-11-20T05:27:02.405 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-11-20T05:27:02.698 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2021-11-20T05:27:03.067 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-20T05:27:03.075 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2021-11-20T05:27:03.323 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-20T05:27:03.433 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-11-20T05:27:03.756 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-20T05:27:03.801 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2021-11-20T05:27:04.020 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-20T05:27:04.030 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-20T05:27:04.252 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-20T05:27:04.479 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-20T05:27:04.496 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-20T05:27:04.644 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-20T05:27:05.029 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-20T05:27:05.029 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2021-11-20T05:27:05.029 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2021-11-20T05:27:05.030 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2021-11-20T05:27:05.030 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2021-11-20T05:27:05.030 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2021-11-20T05:27:05.030 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2021-11-20T05:27:05.030 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2021-11-20T05:27:05.031 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2021-11-20T05:27:05.031 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2021-11-20T05:27:05.031 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2021-11-20T05:27:05.031 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2021-11-20T05:27:05.032 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2021-11-20T05:27:05.032 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2021-11-20T05:27:05.032 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2021-11-20T05:27:05.032 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2021-11-20T05:27:05.032 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2021-11-20T05:27:05.033 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-20T05:27:05.033 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2021-11-20T05:27:05.033 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2021-11-20T05:27:05.033 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2021-11-20T05:27:05.034 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2021-11-20T05:27:05.034 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2021-11-20T05:27:05.034 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2021-11-20T05:27:05.034 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2021-11-20T05:27:05.034 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2021-11-20T05:27:05.035 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2021-11-20T05:27:05.035 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2021-11-20T05:27:05.035 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2021-11-20T05:27:05.036 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2021-11-20T05:27:05.036 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2021-11-20T05:27:05.036 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2021-11-20T05:27:05.641 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-20T05:27:05.878 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-20T05:27:05.996 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2021-11-20T05:27:05.997 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:05.997 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2021-11-20T05:27:05.997 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-2:16.2.4-0.el8.x86_64 2021-11-20T05:27:05.997 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2021-11-20T05:27:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2021-11-20T05:27:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2021-11-20T05:27:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2021-11-20T05:27:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2021-11-20T05:27:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-11-20T05:27:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-11-20T05:27:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-11-20T05:27:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-11-20T05:27:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-11-20T05:27:06.000 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2021-11-20T05:27:06.000 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-6.2-6.el8.noarch 2021-11-20T05:27:06.000 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-11-20T05:27:06.000 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-11-20T05:27:06.001 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako-1.0.6-13.el8.noarch 2021-11-20T05:27:06.001 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-11-20T05:27:06.001 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-11-20T05:27:06.001 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-11-20T05:27:06.001 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend-2.6-1.el8.noarch 2021-11-20T05:27:06.002 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-11-20T05:27:06.002 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser-2.14-14.el8.noarch 2021-11-20T05:27:06.002 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz-2017.2-9.el8.noarch 2021-11-20T05:27:06.002 INFO:teuthology.orchestra.run.smithi187.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-11-20T05:27:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-11-20T05:27:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-11-20T05:27:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-11-20T05:27:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-11-20T05:27:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-11-20T05:27:06.004 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-11-20T05:27:06.004 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-11-20T05:27:06.004 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-11-20T05:27:06.004 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:06.005 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:27:06.232 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-base 2021-11-20T05:27:06.338 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-20T05:27:06.603 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-20T05:27:06.776 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:06 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:06.838 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-20T05:27:07.048 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-20T05:27:07.066 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-20T05:27:07.240 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-20T05:27:07.651 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-20T05:27:07.651 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2021-11-20T05:27:07.652 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2021-11-20T05:27:07.652 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2021-11-20T05:27:07.652 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2021-11-20T05:27:07.652 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2021-11-20T05:27:07.653 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2021-11-20T05:27:07.653 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2021-11-20T05:27:07.653 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2021-11-20T05:27:07.653 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2021-11-20T05:27:07.654 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2021-11-20T05:27:07.654 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2021-11-20T05:27:07.654 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2021-11-20T05:27:07.654 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2021-11-20T05:27:07.655 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2021-11-20T05:27:07.655 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2021-11-20T05:27:07.655 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2021-11-20T05:27:07.655 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-20T05:27:07.655 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2021-11-20T05:27:07.656 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2021-11-20T05:27:07.656 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2021-11-20T05:27:07.657 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2021-11-20T05:27:07.657 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2021-11-20T05:27:07.657 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2021-11-20T05:27:07.657 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2021-11-20T05:27:07.658 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2021-11-20T05:27:07.658 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2021-11-20T05:27:07.658 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2021-11-20T05:27:07.658 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2021-11-20T05:27:07.658 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2021-11-20T05:27:07.659 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2021-11-20T05:27:07.659 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2021-11-20T05:27:07.806 INFO:teuthology.orchestra.run.smithi187.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:27:08.092 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:27:08.094 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2021-11-20T05:27:08.094 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:27:08.163 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install cephadm 2021-11-20T05:27:08.503 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2021-11-20T05:27:08.504 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:08.504 INFO:teuthology.orchestra.run.smithi058.stdout:Installed: 2021-11-20T05:27:08.505 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-2:16.2.4-0.el8.x86_64 2021-11-20T05:27:08.505 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2021-11-20T05:27:08.505 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2021-11-20T05:27:08.506 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2021-11-20T05:27:08.506 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2021-11-20T05:27:08.506 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2021-11-20T05:27:08.507 INFO:teuthology.orchestra.run.smithi058.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-11-20T05:27:08.507 INFO:teuthology.orchestra.run.smithi058.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-11-20T05:27:08.507 INFO:teuthology.orchestra.run.smithi058.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-11-20T05:27:08.508 INFO:teuthology.orchestra.run.smithi058.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-11-20T05:27:08.508 INFO:teuthology.orchestra.run.smithi058.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-11-20T05:27:08.508 INFO:teuthology.orchestra.run.smithi058.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2021-11-20T05:27:08.509 INFO:teuthology.orchestra.run.smithi058.stdout: python3-jaraco-6.2-6.el8.noarch 2021-11-20T05:27:08.509 INFO:teuthology.orchestra.run.smithi058.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-11-20T05:27:08.510 INFO:teuthology.orchestra.run.smithi058.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-11-20T05:27:08.510 INFO:teuthology.orchestra.run.smithi058.stdout: python3-mako-1.0.6-13.el8.noarch 2021-11-20T05:27:08.510 INFO:teuthology.orchestra.run.smithi058.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-11-20T05:27:08.511 INFO:teuthology.orchestra.run.smithi058.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-11-20T05:27:08.511 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-11-20T05:27:08.511 INFO:teuthology.orchestra.run.smithi058.stdout: python3-portend-2.6-1.el8.noarch 2021-11-20T05:27:08.512 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-11-20T05:27:08.512 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pycparser-2.14-14.el8.noarch 2021-11-20T05:27:08.512 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pytz-2017.2-9.el8.noarch 2021-11-20T05:27:08.512 INFO:teuthology.orchestra.run.smithi058.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-11-20T05:27:08.513 INFO:teuthology.orchestra.run.smithi058.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-11-20T05:27:08.513 INFO:teuthology.orchestra.run.smithi058.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-11-20T05:27:08.513 INFO:teuthology.orchestra.run.smithi058.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-11-20T05:27:08.514 INFO:teuthology.orchestra.run.smithi058.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-11-20T05:27:08.514 INFO:teuthology.orchestra.run.smithi058.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-11-20T05:27:08.515 INFO:teuthology.orchestra.run.smithi058.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-11-20T05:27:08.515 INFO:teuthology.orchestra.run.smithi058.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-11-20T05:27:08.515 INFO:teuthology.orchestra.run.smithi058.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-11-20T05:27:08.515 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:08.516 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:27:08.700 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:08 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:08.731 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install ceph-base 2021-11-20T05:27:09.298 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:01:09 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:10.003 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:27:10.004 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:10.004 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2021-11-20T05:27:10.005 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:10.005 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2021-11-20T05:27:10.005 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2021-11-20T05:27:10.005 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:10.005 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:27:10.006 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:10.006 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2021-11-20T05:27:10.006 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:10.006 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 71 k 2021-11-20T05:27:10.007 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 290 k 2021-11-20T05:27:10.007 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2021-11-20T05:27:10.017 INFO:teuthology.orchestra.run.smithi187.stdout:cephadm-16.2.4-0.el8.noarch.rpm 5.6 MB/s | 71 kB 00:00 2021-11-20T05:27:10.018 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:27:10.018 INFO:teuthology.orchestra.run.smithi187.stdout:Total 5.1 MB/s | 71 kB 00:00 2021-11-20T05:27:10.019 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2021-11-20T05:27:10.026 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2021-11-20T05:27:10.026 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2021-11-20T05:27:10.027 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2021-11-20T05:27:10.031 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2021-11-20T05:27:10.101 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2021-11-20T05:27:10.386 INFO:teuthology.orchestra.run.smithi058.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:27:10.689 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:27:10.691 INFO:teuthology.orchestra.run.smithi058.stdout:Nothing to do. 2021-11-20T05:27:10.691 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:27:10.753 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install cephadm 2021-11-20T05:27:11.317 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:01:11 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:12.777 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T05:27:13.075 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T05:27:13.273 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T05:27:13.584 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T05:27:13.585 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:13.585 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2021-11-20T05:27:13.586 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm-2:16.2.4-0.el8.noarch 2021-11-20T05:27:13.586 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:13.587 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:27:13.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-immutable-object-cache 2021-11-20T05:27:13.884 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:27:13.885 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:13.886 INFO:teuthology.orchestra.run.smithi058.stdout: Package Architecture Version Repository Size 2021-11-20T05:27:13.886 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:13.886 INFO:teuthology.orchestra.run.smithi058.stdout:Installing: 2021-11-20T05:27:13.887 INFO:teuthology.orchestra.run.smithi058.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2021-11-20T05:27:13.887 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:13.887 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:27:13.887 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:13.888 INFO:teuthology.orchestra.run.smithi058.stdout:Install 1 Package 2021-11-20T05:27:13.888 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:13.888 INFO:teuthology.orchestra.run.smithi058.stdout:Total download size: 71 k 2021-11-20T05:27:13.888 INFO:teuthology.orchestra.run.smithi058.stdout:Installed size: 290 k 2021-11-20T05:27:13.888 INFO:teuthology.orchestra.run.smithi058.stdout:Downloading Packages: 2021-11-20T05:27:13.898 INFO:teuthology.orchestra.run.smithi058.stdout:cephadm-16.2.4-0.el8.noarch.rpm 5.9 MB/s | 71 kB 00:00 2021-11-20T05:27:13.898 INFO:teuthology.orchestra.run.smithi058.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:27:13.899 INFO:teuthology.orchestra.run.smithi058.stdout:Total 5.3 MB/s | 71 kB 00:00 2021-11-20T05:27:13.899 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction check 2021-11-20T05:27:13.907 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction check succeeded. 2021-11-20T05:27:13.907 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction test 2021-11-20T05:27:13.909 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction test succeeded. 2021-11-20T05:27:13.913 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction 2021-11-20T05:27:13.969 INFO:teuthology.orchestra.run.smithi058.stdout: Preparing : 1/1 2021-11-20T05:27:14.322 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:14 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:15.659 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:27:15.660 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:15.660 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2021-11-20T05:27:15.661 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:15.661 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2021-11-20T05:27:15.661 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2021-11-20T05:27:15.661 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:15.661 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:27:15.662 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:15.662 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2021-11-20T05:27:15.662 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:15.662 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 163 k 2021-11-20T05:27:15.663 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 587 k 2021-11-20T05:27:15.663 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2021-11-20T05:27:15.673 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 13 MB/s | 163 kB 00:00 2021-11-20T05:27:15.674 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:27:15.675 INFO:teuthology.orchestra.run.smithi187.stdout:Total 11 MB/s | 163 kB 00:00 2021-11-20T05:27:15.675 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2021-11-20T05:27:15.687 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2021-11-20T05:27:15.688 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2021-11-20T05:27:15.736 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2021-11-20T05:27:15.740 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2021-11-20T05:27:15.858 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2021-11-20T05:27:16.110 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:27:16.425 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T05:27:16.509 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:27:16.698 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T05:27:16.873 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:27:16.873 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:16.873 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2021-11-20T05:27:16.874 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2021-11-20T05:27:16.874 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:16.874 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:27:16.903 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T05:27:17.078 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr 2021-11-20T05:27:17.214 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T05:27:17.214 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:17.214 INFO:teuthology.orchestra.run.smithi058.stdout:Installed: 2021-11-20T05:27:17.214 INFO:teuthology.orchestra.run.smithi058.stdout: cephadm-2:16.2.4-0.el8.noarch 2021-11-20T05:27:17.215 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:17.215 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:27:17.431 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install ceph-immutable-object-cache 2021-11-20T05:27:17.631 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:17 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:18.007 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:01:18 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:19.561 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:27:19.562 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:19.562 INFO:teuthology.orchestra.run.smithi058.stdout: Package Arch Version Repo Size 2021-11-20T05:27:19.562 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:19.563 INFO:teuthology.orchestra.run.smithi058.stdout:Installing: 2021-11-20T05:27:19.563 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2021-11-20T05:27:19.563 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:19.564 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:27:19.564 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:19.564 INFO:teuthology.orchestra.run.smithi058.stdout:Install 1 Package 2021-11-20T05:27:19.564 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:19.564 INFO:teuthology.orchestra.run.smithi058.stdout:Total download size: 163 k 2021-11-20T05:27:19.565 INFO:teuthology.orchestra.run.smithi058.stdout:Installed size: 587 k 2021-11-20T05:27:19.565 INFO:teuthology.orchestra.run.smithi058.stdout:Downloading Packages: 2021-11-20T05:27:19.577 INFO:teuthology.orchestra.run.smithi058.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 12 MB/s | 163 kB 00:00 2021-11-20T05:27:19.577 INFO:teuthology.orchestra.run.smithi058.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:27:19.578 INFO:teuthology.orchestra.run.smithi058.stdout:Total 11 MB/s | 163 kB 00:00 2021-11-20T05:27:19.579 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction check 2021-11-20T05:27:19.591 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction check succeeded. 2021-11-20T05:27:19.592 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction test 2021-11-20T05:27:19.644 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction test succeeded. 2021-11-20T05:27:19.647 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction 2021-11-20T05:27:19.721 INFO:teuthology.orchestra.run.smithi187.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:27:19.787 INFO:teuthology.orchestra.run.smithi058.stdout: Preparing : 1/1 2021-11-20T05:27:20.010 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:27:20.011 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2021-11-20T05:27:20.011 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:27:20.030 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:27:20.129 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-dashboard 2021-11-20T05:27:20.430 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:27:20.669 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:20 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:20.813 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:27:20.813 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:20.813 INFO:teuthology.orchestra.run.smithi058.stdout:Installed: 2021-11-20T05:27:20.813 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2021-11-20T05:27:20.814 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:20.814 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:27:21.007 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install ceph-mgr 2021-11-20T05:27:21.577 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:01:21 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:21.984 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:27:21.985 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:21.985 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2021-11-20T05:27:21.985 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:21.985 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2021-11-20T05:27:21.985 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2021-11-20T05:27:21.986 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2021-11-20T05:27:21.986 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2021-11-20T05:27:21.986 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2021-11-20T05:27:21.987 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-11-20T05:27:21.987 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-11-20T05:27:21.987 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-11-20T05:27:21.987 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:21.988 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:27:21.988 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:21.988 INFO:teuthology.orchestra.run.smithi187.stdout:Install 6 Packages 2021-11-20T05:27:21.988 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:21.988 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 3.0 M 2021-11-20T05:27:21.989 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 15 M 2021-11-20T05:27:21.989 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2021-11-20T05:27:22.006 INFO:teuthology.orchestra.run.smithi187.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 1.2 MB/s | 24 kB 00:00 2021-11-20T05:27:22.013 INFO:teuthology.orchestra.run.smithi187.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 333 kB/s | 8.9 kB 00:00 2021-11-20T05:27:22.042 INFO:teuthology.orchestra.run.smithi187.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 48 MB/s | 2.7 MB 00:00 2021-11-20T05:27:22.045 INFO:teuthology.orchestra.run.smithi187.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.1 MB/s | 43 kB 00:00 2021-11-20T05:27:22.096 INFO:teuthology.orchestra.run.smithi187.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 406 kB/s | 34 kB 00:00 2021-11-20T05:27:22.182 INFO:teuthology.orchestra.run.smithi187.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2021-11-20T05:27:22.185 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:27:22.185 INFO:teuthology.orchestra.run.smithi187.stdout:Total 15 MB/s | 3.0 MB 00:00 2021-11-20T05:27:22.188 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2021-11-20T05:27:22.197 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2021-11-20T05:27:22.197 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2021-11-20T05:27:22.270 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2021-11-20T05:27:22.273 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2021-11-20T05:27:22.542 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2021-11-20T05:27:22.701 INFO:teuthology.orchestra.run.smithi058.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:27:22.851 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-11-20T05:27:23.004 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:27:23.006 INFO:teuthology.orchestra.run.smithi058.stdout:Nothing to do. 2021-11-20T05:27:23.006 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:27:23.066 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install ceph-mgr-dashboard 2021-11-20T05:27:23.105 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-11-20T05:27:23.311 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-11-20T05:27:23.424 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2021-11-20T05:27:23.635 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:01:23 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:24.488 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2021-11-20T05:27:24.852 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-20T05:27:25.093 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-20T05:27:25.093 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2021-11-20T05:27:25.094 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2021-11-20T05:27:25.094 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2021-11-20T05:27:25.094 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-11-20T05:27:25.094 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-11-20T05:27:25.563 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:27:25.564 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:25.564 INFO:teuthology.orchestra.run.smithi058.stdout: Package Arch Version Repository Size 2021-11-20T05:27:25.564 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:25.565 INFO:teuthology.orchestra.run.smithi058.stdout:Installing: 2021-11-20T05:27:25.565 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2021-11-20T05:27:25.565 INFO:teuthology.orchestra.run.smithi058.stdout:Installing dependencies: 2021-11-20T05:27:25.565 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2021-11-20T05:27:25.566 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2021-11-20T05:27:25.566 INFO:teuthology.orchestra.run.smithi058.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-11-20T05:27:25.566 INFO:teuthology.orchestra.run.smithi058.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-11-20T05:27:25.566 INFO:teuthology.orchestra.run.smithi058.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-11-20T05:27:25.567 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:25.567 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:27:25.567 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:25.567 INFO:teuthology.orchestra.run.smithi058.stdout:Install 6 Packages 2021-11-20T05:27:25.567 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:25.568 INFO:teuthology.orchestra.run.smithi058.stdout:Total download size: 3.0 M 2021-11-20T05:27:25.569 INFO:teuthology.orchestra.run.smithi058.stdout:Installed size: 15 M 2021-11-20T05:27:25.569 INFO:teuthology.orchestra.run.smithi058.stdout:Downloading Packages: 2021-11-20T05:27:25.588 INFO:teuthology.orchestra.run.smithi058.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 1.1 MB/s | 24 kB 00:00 2021-11-20T05:27:25.589 INFO:teuthology.orchestra.run.smithi058.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 396 kB/s | 8.9 kB 00:00 2021-11-20T05:27:25.618 INFO:teuthology.orchestra.run.smithi058.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 52 MB/s | 2.7 MB 00:00 2021-11-20T05:27:25.622 INFO:teuthology.orchestra.run.smithi058.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.2 MB/s | 43 kB 00:00 2021-11-20T05:27:25.678 INFO:teuthology.orchestra.run.smithi058.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 412 kB/s | 34 kB 00:00 2021-11-20T05:27:25.719 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-11-20T05:27:25.720 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:25.720 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2021-11-20T05:27:25.720 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2021-11-20T05:27:25.720 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2021-11-20T05:27:25.721 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2021-11-20T05:27:25.721 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-11-20T05:27:25.721 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-11-20T05:27:25.721 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes-2.4.1-12.el8.noarch 2021-11-20T05:27:25.721 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:25.722 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:27:25.755 INFO:teuthology.orchestra.run.smithi058.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2021-11-20T05:27:25.758 INFO:teuthology.orchestra.run.smithi058.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:27:25.759 INFO:teuthology.orchestra.run.smithi058.stdout:Total 15 MB/s | 3.0 MB 00:00 2021-11-20T05:27:25.761 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction check 2021-11-20T05:27:25.771 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction check succeeded. 2021-11-20T05:27:25.771 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction test 2021-11-20T05:27:25.848 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction test succeeded. 2021-11-20T05:27:25.852 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction 2021-11-20T05:27:25.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-diskprediction-local 2021-11-20T05:27:26.133 INFO:teuthology.orchestra.run.smithi058.stdout: Preparing : 1/1 2021-11-20T05:27:26.424 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-11-20T05:27:26.510 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:26 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:26.676 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-11-20T05:27:26.880 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-11-20T05:27:27.003 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2021-11-20T05:27:27.852 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:27:27.853 INFO:teuthology.orchestra.run.smithi187.stdout:===================================================================================================== 2021-11-20T05:27:27.854 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2021-11-20T05:27:27.854 INFO:teuthology.orchestra.run.smithi187.stdout:===================================================================================================== 2021-11-20T05:27:27.854 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2021-11-20T05:27:27.855 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2021-11-20T05:27:27.855 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2021-11-20T05:27:27.855 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran x86_64 8.5.0-3.el8 CentOS-Base 643 k 2021-11-20T05:27:27.855 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath x86_64 8.5.0-3.el8 CentOS-Base 170 k 2021-11-20T05:27:27.855 INFO:teuthology.orchestra.run.smithi187.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2021-11-20T05:27:27.856 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2021-11-20T05:27:27.856 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-11-20T05:27:27.856 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-11-20T05:27:27.856 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2021-11-20T05:27:27.857 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:27.857 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:27:27.857 INFO:teuthology.orchestra.run.smithi187.stdout:===================================================================================================== 2021-11-20T05:27:27.857 INFO:teuthology.orchestra.run.smithi187.stdout:Install 8 Packages 2021-11-20T05:27:27.857 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:27.858 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 37 M 2021-11-20T05:27:27.859 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 211 M 2021-11-20T05:27:27.859 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2021-11-20T05:27:27.956 INFO:teuthology.orchestra.run.smithi187.stdout:(1/8): openblas-0.3.15-3.el8.x86_64.rpm 48 MB/s | 4.7 MB 00:00 2021-11-20T05:27:27.986 INFO:teuthology.orchestra.run.smithi187.stdout:(2/8): ceph-mgr-diskprediction-local-16.2.4-0.e 66 MB/s | 8.6 MB 00:00 2021-11-20T05:27:28.034 INFO:teuthology.orchestra.run.smithi187.stdout:(3/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 48 MB/s | 3.7 MB 00:00 2021-11-20T05:27:28.056 INFO:teuthology.orchestra.run.smithi187.stdout:(4/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 3.1 MB/s | 225 kB 00:00 2021-11-20T05:27:28.151 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2021-11-20T05:27:28.162 INFO:teuthology.orchestra.run.smithi187.stdout:(5/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 16 MB/s | 4.9 MB 00:00 2021-11-20T05:27:28.179 INFO:teuthology.orchestra.run.smithi187.stdout:(6/8): libgfortran-8.5.0-3.el8.x86_64.rpm 5.1 MB/s | 643 kB 00:00 2021-11-20T05:27:28.180 INFO:teuthology.orchestra.run.smithi187.stdout:(7/8): libquadmath-8.5.0-3.el8.x86_64.rpm 9.2 MB/s | 170 kB 00:00 2021-11-20T05:27:28.313 INFO:teuthology.orchestra.run.smithi187.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 49 MB/s | 14 MB 00:00 2021-11-20T05:27:28.317 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:27:28.317 INFO:teuthology.orchestra.run.smithi187.stdout:Total 79 MB/s | 37 MB 00:00 2021-11-20T05:27:28.332 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2021-11-20T05:27:28.351 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2021-11-20T05:27:28.352 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2021-11-20T05:27:28.441 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-20T05:27:28.656 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2021-11-20T05:27:28.660 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2021-11-20T05:27:28.742 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-20T05:27:28.742 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2021-11-20T05:27:28.743 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2021-11-20T05:27:28.743 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2021-11-20T05:27:28.743 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-11-20T05:27:28.743 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-11-20T05:27:29.126 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-11-20T05:27:29.127 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:29.127 INFO:teuthology.orchestra.run.smithi058.stdout:Installed: 2021-11-20T05:27:29.127 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2021-11-20T05:27:29.128 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2021-11-20T05:27:29.128 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2021-11-20T05:27:29.128 INFO:teuthology.orchestra.run.smithi058.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-11-20T05:27:29.128 INFO:teuthology.orchestra.run.smithi058.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-11-20T05:27:29.128 INFO:teuthology.orchestra.run.smithi058.stdout: python3-routes-2.4.1-12.el8.noarch 2021-11-20T05:27:29.129 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:29.129 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:27:29.214 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install ceph-mgr-diskprediction-local 2021-11-20T05:27:29.229 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2021-11-20T05:27:29.424 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-20T05:27:29.555 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-20T05:27:29.707 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-20T05:27:29.784 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:01:29 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:30.302 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-20T05:27:30.457 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2021-11-20T05:27:31.072 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2021-11-20T05:27:31.133 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:27:31.134 INFO:teuthology.orchestra.run.smithi058.stdout:===================================================================================================== 2021-11-20T05:27:31.134 INFO:teuthology.orchestra.run.smithi058.stdout: Package Arch Version Repository Size 2021-11-20T05:27:31.135 INFO:teuthology.orchestra.run.smithi058.stdout:===================================================================================================== 2021-11-20T05:27:31.135 INFO:teuthology.orchestra.run.smithi058.stdout:Installing: 2021-11-20T05:27:31.135 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2021-11-20T05:27:31.135 INFO:teuthology.orchestra.run.smithi058.stdout:Installing dependencies: 2021-11-20T05:27:31.136 INFO:teuthology.orchestra.run.smithi058.stdout: libgfortran x86_64 8.5.0-3.el8 CentOS-Base 643 k 2021-11-20T05:27:31.136 INFO:teuthology.orchestra.run.smithi058.stdout: libquadmath x86_64 8.5.0-3.el8 CentOS-Base 170 k 2021-11-20T05:27:31.136 INFO:teuthology.orchestra.run.smithi058.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2021-11-20T05:27:31.137 INFO:teuthology.orchestra.run.smithi058.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2021-11-20T05:27:31.137 INFO:teuthology.orchestra.run.smithi058.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-11-20T05:27:31.137 INFO:teuthology.orchestra.run.smithi058.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-11-20T05:27:31.137 INFO:teuthology.orchestra.run.smithi058.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2021-11-20T05:27:31.137 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:31.138 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:27:31.138 INFO:teuthology.orchestra.run.smithi058.stdout:===================================================================================================== 2021-11-20T05:27:31.138 INFO:teuthology.orchestra.run.smithi058.stdout:Install 8 Packages 2021-11-20T05:27:31.138 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:31.139 INFO:teuthology.orchestra.run.smithi058.stdout:Total download size: 37 M 2021-11-20T05:27:31.140 INFO:teuthology.orchestra.run.smithi058.stdout:Installed size: 211 M 2021-11-20T05:27:31.140 INFO:teuthology.orchestra.run.smithi058.stdout:Downloading Packages: 2021-11-20T05:27:31.274 INFO:teuthology.orchestra.run.smithi058.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 63 MB/s | 8.6 MB 00:00 2021-11-20T05:27:31.288 INFO:teuthology.orchestra.run.smithi058.stdout:(2/8): openblas-0.3.15-3.el8.x86_64.rpm 31 MB/s | 4.7 MB 00:00 2021-11-20T05:27:31.300 INFO:teuthology.orchestra.run.smithi058.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 35 MB/s | 225 kB 00:00 2021-11-20T05:27:31.361 INFO:teuthology.orchestra.run.smithi058.stdout:(4/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 22 MB/s | 4.9 MB 00:00 2021-11-20T05:27:31.376 INFO:teuthology.orchestra.run.smithi058.stdout:(5/8): libgfortran-8.5.0-3.el8.x86_64.rpm 42 MB/s | 643 kB 00:00 2021-11-20T05:27:31.382 INFO:teuthology.orchestra.run.smithi058.stdout:(6/8): libquadmath-8.5.0-3.el8.x86_64.rpm 29 MB/s | 170 kB 00:00 2021-11-20T05:27:31.398 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-20T05:27:31.456 INFO:teuthology.orchestra.run.smithi058.stdout:(7/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 21 MB/s | 3.7 MB 00:00 2021-11-20T05:27:31.620 INFO:teuthology.orchestra.run.smithi058.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 43 MB/s | 14 MB 00:00 2021-11-20T05:27:31.625 INFO:teuthology.orchestra.run.smithi058.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:27:31.625 INFO:teuthology.orchestra.run.smithi058.stdout:Total 75 MB/s | 37 MB 00:00 2021-11-20T05:27:31.641 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction check 2021-11-20T05:27:31.662 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction check succeeded. 2021-11-20T05:27:31.663 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction test 2021-11-20T05:27:31.985 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction test succeeded. 2021-11-20T05:27:31.989 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction 2021-11-20T05:27:32.112 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-20T05:27:32.509 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-11-20T05:27:32.582 INFO:teuthology.orchestra.run.smithi058.stdout: Preparing : 1/1 2021-11-20T05:27:32.794 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-20T05:27:32.798 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-20T05:27:32.945 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-20T05:27:33.085 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-20T05:27:33.757 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-20T05:27:33.952 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2021-11-20T05:27:34.608 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2021-11-20T05:27:34.611 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-20T05:27:34.769 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-20T05:27:35.519 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-20T05:27:35.950 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2021-11-20T05:27:36.852 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-11-20T05:27:36.965 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-20T05:27:37.101 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-20T05:27:37.274 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-20T05:27:37.274 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2021-11-20T05:27:37.274 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2021-11-20T05:27:37.275 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2021-11-20T05:27:37.275 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-11-20T05:27:37.275 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-11-20T05:27:37.275 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2021-11-20T05:27:37.276 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libgfortran-8.5.0-3.el8.x86_64 7/8 2021-11-20T05:27:37.620 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libquadmath-8.5.0-3.el8.x86_64 8/8 2021-11-20T05:27:37.621 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:37.621 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2021-11-20T05:27:37.621 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2021-11-20T05:27:37.622 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran-8.5.0-3.el8.x86_64 2021-11-20T05:27:37.622 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath-8.5.0-3.el8.x86_64 2021-11-20T05:27:37.623 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-0.3.15-3.el8.x86_64 2021-11-20T05:27:37.623 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-threads-0.3.15-3.el8.x86_64 2021-11-20T05:27:37.623 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-11-20T05:27:37.624 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-11-20T05:27:37.624 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2021-11-20T05:27:37.624 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:37.625 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:27:37.738 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-rook 2021-11-20T05:27:38.278 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:38 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:39.037 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-20T05:27:39.581 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:27:39.582 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:39.582 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2021-11-20T05:27:39.583 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:39.583 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2021-11-20T05:27:39.583 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2021-11-20T05:27:39.583 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2021-11-20T05:27:39.583 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-11-20T05:27:39.584 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-11-20T05:27:39.584 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-11-20T05:27:39.584 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-11-20T05:27:39.584 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-11-20T05:27:39.585 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-11-20T05:27:39.585 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-11-20T05:27:39.585 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-11-20T05:27:39.586 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-11-20T05:27:39.586 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-11-20T05:27:39.586 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa noarch 4.7.2-1.el8 epel 61 k 2021-11-20T05:27:39.586 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-11-20T05:27:39.586 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:39.587 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:27:39.587 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:39.588 INFO:teuthology.orchestra.run.smithi187.stdout:Install 13 Packages 2021-11-20T05:27:39.588 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:39.588 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 1.9 M 2021-11-20T05:27:39.589 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 24 M 2021-11-20T05:27:39.589 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2021-11-20T05:27:39.604 INFO:teuthology.orchestra.run.smithi187.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 27 kB 00:00 2021-11-20T05:27:39.621 INFO:teuthology.orchestra.run.smithi187.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 757 kB/s | 27 kB 00:00 2021-11-20T05:27:39.623 INFO:teuthology.orchestra.run.smithi187.stdout:(3/13): python3-pyasn1-modules-0.3.7-6.el8.noar 45 MB/s | 110 kB 00:00 2021-11-20T05:27:39.626 INFO:teuthology.orchestra.run.smithi187.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 57 MB/s | 155 kB 00:00 2021-11-20T05:27:39.627 INFO:teuthology.orchestra.run.smithi187.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 31 MB/s | 43 kB 00:00 2021-11-20T05:27:39.633 INFO:teuthology.orchestra.run.smithi187.stdout:(6/13): python3-jsonpointer-1.10-11.el8.noarch. 421 kB/s | 20 kB 00:00 2021-11-20T05:27:39.645 INFO:teuthology.orchestra.run.smithi187.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.1 MB/s | 126 kB 00:00 2021-11-20T05:27:39.713 INFO:teuthology.orchestra.run.smithi187.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 389 kB/s | 33 kB 00:00 2021-11-20T05:27:39.714 INFO:teuthology.orchestra.run.smithi187.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 201 kB/s | 16 kB 00:00 2021-11-20T05:27:39.755 INFO:teuthology.orchestra.run.smithi187.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 736 kB/s | 81 kB 00:00 2021-11-20T05:27:39.769 INFO:teuthology.orchestra.run.smithi187.stdout:(11/13): python3-rsa-4.7.2-1.el8.noarch.rpm 1.1 MB/s | 61 kB 00:00 2021-11-20T05:27:39.784 INFO:teuthology.orchestra.run.smithi187.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 2.1 MB/s | 61 kB 00:00 2021-11-20T05:27:39.883 INFO:teuthology.orchestra.run.smithi187.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 6.9 MB/s | 1.2 MB 00:00 2021-11-20T05:27:39.886 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:27:39.887 INFO:teuthology.orchestra.run.smithi187.stdout:Total 6.3 MB/s | 1.9 MB 00:00 2021-11-20T05:27:39.897 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2021-11-20T05:27:39.910 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2021-11-20T05:27:39.911 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2021-11-20T05:27:39.987 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2021-11-20T05:27:39.990 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2021-11-20T05:27:40.442 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2021-11-20T05:27:40.524 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2021-11-20T05:27:40.651 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-20T05:27:40.710 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-11-20T05:27:40.916 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-11-20T05:27:40.956 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-20T05:27:40.956 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2021-11-20T05:27:40.957 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2021-11-20T05:27:40.957 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2021-11-20T05:27:40.957 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-11-20T05:27:40.957 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-11-20T05:27:40.958 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2021-11-20T05:27:40.958 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : libgfortran-8.5.0-3.el8.x86_64 7/8 2021-11-20T05:27:41.175 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rsa-4.7.2-1.el8.noarch 3/13 2021-11-20T05:27:41.318 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : libquadmath-8.5.0-3.el8.x86_64 8/8 2021-11-20T05:27:41.318 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:41.318 INFO:teuthology.orchestra.run.smithi058.stdout:Installed: 2021-11-20T05:27:41.319 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2021-11-20T05:27:41.319 INFO:teuthology.orchestra.run.smithi058.stdout: libgfortran-8.5.0-3.el8.x86_64 2021-11-20T05:27:41.319 INFO:teuthology.orchestra.run.smithi058.stdout: libquadmath-8.5.0-3.el8.x86_64 2021-11-20T05:27:41.319 INFO:teuthology.orchestra.run.smithi058.stdout: openblas-0.3.15-3.el8.x86_64 2021-11-20T05:27:41.319 INFO:teuthology.orchestra.run.smithi058.stdout: openblas-threads-0.3.15-3.el8.x86_64 2021-11-20T05:27:41.320 INFO:teuthology.orchestra.run.smithi058.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-11-20T05:27:41.320 INFO:teuthology.orchestra.run.smithi058.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-11-20T05:27:41.320 INFO:teuthology.orchestra.run.smithi058.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2021-11-20T05:27:41.320 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:41.321 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:27:41.395 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-11-20T05:27:41.413 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install ceph-mgr-rook 2021-11-20T05:27:41.628 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-11-20T05:27:41.913 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-11-20T05:27:41.983 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:01:41 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:42.210 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-11-20T05:27:42.496 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-11-20T05:27:43.351 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:27:43.353 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:43.353 INFO:teuthology.orchestra.run.smithi058.stdout: Package Arch Version Repository Size 2021-11-20T05:27:43.354 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:43.354 INFO:teuthology.orchestra.run.smithi058.stdout:Installing: 2021-11-20T05:27:43.354 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2021-11-20T05:27:43.354 INFO:teuthology.orchestra.run.smithi058.stdout:Installing dependencies: 2021-11-20T05:27:43.355 INFO:teuthology.orchestra.run.smithi058.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-11-20T05:27:43.355 INFO:teuthology.orchestra.run.smithi058.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-11-20T05:27:43.355 INFO:teuthology.orchestra.run.smithi058.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-11-20T05:27:43.356 INFO:teuthology.orchestra.run.smithi058.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-11-20T05:27:43.356 INFO:teuthology.orchestra.run.smithi058.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-11-20T05:27:43.356 INFO:teuthology.orchestra.run.smithi058.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-11-20T05:27:43.356 INFO:teuthology.orchestra.run.smithi058.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-11-20T05:27:43.357 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-11-20T05:27:43.357 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-11-20T05:27:43.357 INFO:teuthology.orchestra.run.smithi058.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-11-20T05:27:43.357 INFO:teuthology.orchestra.run.smithi058.stdout: python3-rsa noarch 4.7.2-1.el8 epel 61 k 2021-11-20T05:27:43.358 INFO:teuthology.orchestra.run.smithi058.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-11-20T05:27:43.358 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:43.358 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:27:43.358 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:43.359 INFO:teuthology.orchestra.run.smithi058.stdout:Install 13 Packages 2021-11-20T05:27:43.359 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:43.360 INFO:teuthology.orchestra.run.smithi058.stdout:Total download size: 1.9 M 2021-11-20T05:27:43.360 INFO:teuthology.orchestra.run.smithi058.stdout:Installed size: 24 M 2021-11-20T05:27:43.360 INFO:teuthology.orchestra.run.smithi058.stdout:Downloading Packages: 2021-11-20T05:27:43.369 INFO:teuthology.orchestra.run.smithi058.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 2.3 MB/s | 27 kB 00:00 2021-11-20T05:27:43.481 INFO:teuthology.orchestra.run.smithi058.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 214 kB/s | 27 kB 00:00 2021-11-20T05:27:43.487 INFO:teuthology.orchestra.run.smithi058.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 153 kB/s | 20 kB 00:00 2021-11-20T05:27:43.528 INFO:teuthology.orchestra.run.smithi058.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.3 MB/s | 110 kB 00:00 2021-11-20T05:27:43.552 INFO:teuthology.orchestra.run.smithi058.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 688 kB/s | 126 kB 00:00 2021-11-20T05:27:43.553 INFO:teuthology.orchestra.run.smithi058.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.7 MB/s | 43 kB 00:00 2021-11-20T05:27:43.558 INFO:teuthology.orchestra.run.smithi058.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.1 MB/s | 155 kB 00:00 2021-11-20T05:27:43.604 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-11-20T05:27:43.640 INFO:teuthology.orchestra.run.smithi058.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 188 kB/s | 16 kB 00:00 2021-11-20T05:27:43.642 INFO:teuthology.orchestra.run.smithi058.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 367 kB/s | 33 kB 00:00 2021-11-20T05:27:43.672 INFO:teuthology.orchestra.run.smithi058.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 716 kB/s | 81 kB 00:00 2021-11-20T05:27:43.699 INFO:teuthology.orchestra.run.smithi058.stdout:(11/13): python3-rsa-4.7.2-1.el8.noarch.rpm 1.1 MB/s | 61 kB 00:00 2021-11-20T05:27:43.700 INFO:teuthology.orchestra.run.smithi058.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 2.1 MB/s | 61 kB 00:00 2021-11-20T05:27:43.839 INFO:teuthology.orchestra.run.smithi058.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.9 MB/s | 1.2 MB 00:00 2021-11-20T05:27:43.843 INFO:teuthology.orchestra.run.smithi058.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:27:43.843 INFO:teuthology.orchestra.run.smithi058.stdout:Total 3.9 MB/s | 1.9 MB 00:00 2021-11-20T05:27:43.854 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction check 2021-11-20T05:27:43.870 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction check succeeded. 2021-11-20T05:27:43.870 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction test 2021-11-20T05:27:43.904 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-11-20T05:27:43.954 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction test succeeded. 2021-11-20T05:27:43.958 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction 2021-11-20T05:27:44.103 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-11-20T05:27:44.312 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-11-20T05:27:44.483 INFO:teuthology.orchestra.run.smithi058.stdout: Preparing : 1/1 2021-11-20T05:27:44.496 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-20T05:27:44.675 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-11-20T05:27:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-20T05:27:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2021-11-20T05:27:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-11-20T05:27:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-11-20T05:27:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-11-20T05:27:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-11-20T05:27:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-11-20T05:27:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-11-20T05:27:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-11-20T05:27:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-11-20T05:27:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-11-20T05:27:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-11-20T05:27:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rsa-4.7.2-1.el8.noarch 12/13 2021-11-20T05:27:44.889 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-11-20T05:27:45.139 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-rsa-4.7.2-1.el8.noarch 3/13 2021-11-20T05:27:45.143 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-11-20T05:27:45.143 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:45.144 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2021-11-20T05:27:45.144 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2021-11-20T05:27:45.144 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-11-20T05:27:45.144 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-11-20T05:27:45.145 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-11-20T05:27:45.145 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-11-20T05:27:45.145 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-11-20T05:27:45.145 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-11-20T05:27:45.145 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-11-20T05:27:45.146 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-11-20T05:27:45.146 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-11-20T05:27:45.146 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-11-20T05:27:45.146 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa-4.7.2-1.el8.noarch 2021-11-20T05:27:45.146 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-11-20T05:27:45.147 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:45.147 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:27:45.248 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-cephadm 2021-11-20T05:27:45.358 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-11-20T05:27:45.594 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-11-20T05:27:45.787 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:45 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:45.878 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-11-20T05:27:46.198 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-11-20T05:27:46.454 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-11-20T05:27:47.131 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:27:47.132 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:47.132 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2021-11-20T05:27:47.132 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:47.133 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2021-11-20T05:27:47.133 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2021-11-20T05:27:47.133 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2021-11-20T05:27:47.133 INFO:teuthology.orchestra.run.smithi187.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-11-20T05:27:47.134 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2021-11-20T05:27:47.134 INFO:teuthology.orchestra.run.smithi187.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-11-20T05:27:47.134 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2021-11-20T05:27:47.134 INFO:teuthology.orchestra.run.smithi187.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2021-11-20T05:27:47.134 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:47.135 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:27:47.135 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:47.135 INFO:teuthology.orchestra.run.smithi187.stdout:Install 6 Packages 2021-11-20T05:27:47.135 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:47.136 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 5.8 M 2021-11-20T05:27:47.136 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 24 M 2021-11-20T05:27:47.137 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2021-11-20T05:27:47.154 INFO:teuthology.orchestra.run.smithi187.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 4.5 MB/s | 89 kB 00:00 2021-11-20T05:27:47.191 INFO:teuthology.orchestra.run.smithi187.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 9.3 MB/s | 538 kB 00:00 2021-11-20T05:27:47.231 INFO:teuthology.orchestra.run.smithi187.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 50 MB/s | 4.8 MB 00:00 2021-11-20T05:27:47.240 INFO:teuthology.orchestra.run.smithi187.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 246 kB/s | 21 kB 00:00 2021-11-20T05:27:47.401 INFO:teuthology.orchestra.run.smithi187.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 289 kB/s | 49 kB 00:00 2021-11-20T05:27:47.475 INFO:teuthology.orchestra.run.smithi187.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.2 MB/s | 355 kB 00:00 2021-11-20T05:27:47.478 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:27:47.479 INFO:teuthology.orchestra.run.smithi187.stdout:Total 17 MB/s | 5.8 MB 00:00 2021-11-20T05:27:47.484 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2021-11-20T05:27:47.493 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2021-11-20T05:27:47.493 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2021-11-20T05:27:47.579 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2021-11-20T05:27:47.582 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2021-11-20T05:27:47.606 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-11-20T05:27:47.875 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-11-20T05:27:47.951 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2021-11-20T05:27:48.117 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-11-20T05:27:48.225 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-11-20T05:27:48.334 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-11-20T05:27:48.525 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-20T05:27:48.526 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-11-20T05:27:48.787 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-20T05:27:48.787 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2021-11-20T05:27:48.788 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-11-20T05:27:48.788 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-11-20T05:27:48.788 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-11-20T05:27:48.788 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-11-20T05:27:48.789 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-11-20T05:27:48.789 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-11-20T05:27:48.789 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-11-20T05:27:48.789 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-11-20T05:27:48.790 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-11-20T05:27:48.790 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-11-20T05:27:48.790 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-rsa-4.7.2-1.el8.noarch 12/13 2021-11-20T05:27:49.225 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-11-20T05:27:49.226 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:49.226 INFO:teuthology.orchestra.run.smithi058.stdout:Installed: 2021-11-20T05:27:49.226 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2021-11-20T05:27:49.226 INFO:teuthology.orchestra.run.smithi058.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-11-20T05:27:49.227 INFO:teuthology.orchestra.run.smithi058.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-11-20T05:27:49.227 INFO:teuthology.orchestra.run.smithi058.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-11-20T05:27:49.227 INFO:teuthology.orchestra.run.smithi058.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-11-20T05:27:49.227 INFO:teuthology.orchestra.run.smithi058.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-11-20T05:27:49.228 INFO:teuthology.orchestra.run.smithi058.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-11-20T05:27:49.228 INFO:teuthology.orchestra.run.smithi058.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-11-20T05:27:49.228 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-11-20T05:27:49.228 INFO:teuthology.orchestra.run.smithi058.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-11-20T05:27:49.228 INFO:teuthology.orchestra.run.smithi058.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-11-20T05:27:49.229 INFO:teuthology.orchestra.run.smithi058.stdout: python3-rsa-4.7.2-1.el8.noarch 2021-11-20T05:27:49.229 INFO:teuthology.orchestra.run.smithi058.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-11-20T05:27:49.229 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:49.229 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:27:49.329 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install ceph-mgr-cephadm 2021-11-20T05:27:49.496 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2021-11-20T05:27:49.902 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:01:49 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:49.902 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2021-11-20T05:27:50.193 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2021-11-20T05:27:50.387 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-20T05:27:50.678 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-20T05:27:50.678 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2021-11-20T05:27:50.679 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2021-11-20T05:27:50.679 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2021-11-20T05:27:50.680 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-11-20T05:27:50.680 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-11-20T05:27:51.075 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2021-11-20T05:27:51.075 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:51.076 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2021-11-20T05:27:51.076 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2021-11-20T05:27:51.076 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2021-11-20T05:27:51.077 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2021-11-20T05:27:51.077 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:51.078 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:27:51.181 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-fuse 2021-11-20T05:27:51.270 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:27:51.271 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:51.271 INFO:teuthology.orchestra.run.smithi058.stdout: Package Arch Version Repository Size 2021-11-20T05:27:51.271 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:51.272 INFO:teuthology.orchestra.run.smithi058.stdout:Installing: 2021-11-20T05:27:51.272 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2021-11-20T05:27:51.272 INFO:teuthology.orchestra.run.smithi058.stdout:Installing dependencies: 2021-11-20T05:27:51.272 INFO:teuthology.orchestra.run.smithi058.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-11-20T05:27:51.273 INFO:teuthology.orchestra.run.smithi058.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2021-11-20T05:27:51.273 INFO:teuthology.orchestra.run.smithi058.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-11-20T05:27:51.273 INFO:teuthology.orchestra.run.smithi058.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2021-11-20T05:27:51.273 INFO:teuthology.orchestra.run.smithi058.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2021-11-20T05:27:51.273 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:51.274 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:27:51.274 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:51.274 INFO:teuthology.orchestra.run.smithi058.stdout:Install 6 Packages 2021-11-20T05:27:51.274 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:51.275 INFO:teuthology.orchestra.run.smithi058.stdout:Total download size: 5.8 M 2021-11-20T05:27:51.275 INFO:teuthology.orchestra.run.smithi058.stdout:Installed size: 24 M 2021-11-20T05:27:51.276 INFO:teuthology.orchestra.run.smithi058.stdout:Downloading Packages: 2021-11-20T05:27:51.294 INFO:teuthology.orchestra.run.smithi058.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 4.2 MB/s | 89 kB 00:00 2021-11-20T05:27:51.340 INFO:teuthology.orchestra.run.smithi058.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 7.9 MB/s | 538 kB 00:00 2021-11-20T05:27:51.374 INFO:teuthology.orchestra.run.smithi058.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 48 MB/s | 4.8 MB 00:00 2021-11-20T05:27:51.378 INFO:teuthology.orchestra.run.smithi058.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 253 kB/s | 21 kB 00:00 2021-11-20T05:27:51.485 INFO:teuthology.orchestra.run.smithi058.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 440 kB/s | 49 kB 00:00 2021-11-20T05:27:51.514 INFO:teuthology.orchestra.run.smithi058.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.0 MB/s | 355 kB 00:00 2021-11-20T05:27:51.516 INFO:teuthology.orchestra.run.smithi058.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:27:51.517 INFO:teuthology.orchestra.run.smithi058.stdout:Total 24 MB/s | 5.8 MB 00:00 2021-11-20T05:27:51.522 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction check 2021-11-20T05:27:51.532 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction check succeeded. 2021-11-20T05:27:51.532 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction test 2021-11-20T05:27:51.622 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction test succeeded. 2021-11-20T05:27:51.626 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction 2021-11-20T05:27:51.720 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:51 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:51.934 INFO:teuthology.orchestra.run.smithi058.stdout: Preparing : 1/1 2021-11-20T05:27:52.220 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-11-20T05:27:52.491 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-11-20T05:27:52.996 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:27:52.998 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:52.998 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2021-11-20T05:27:52.999 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:52.999 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2021-11-20T05:27:53.000 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2021-11-20T05:27:53.000 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2021-11-20T05:27:53.001 INFO:teuthology.orchestra.run.smithi187.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-11-20T05:27:53.002 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:53.002 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:27:53.002 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:53.003 INFO:teuthology.orchestra.run.smithi187.stdout:Install 2 Packages 2021-11-20T05:27:53.003 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:53.004 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 967 k 2021-11-20T05:27:53.004 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 3.2 M 2021-11-20T05:27:53.005 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2021-11-20T05:27:53.020 INFO:teuthology.orchestra.run.smithi187.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 41 MB/s | 885 kB 00:00 2021-11-20T05:27:53.194 INFO:teuthology.orchestra.run.smithi187.stdout:(2/2): fuse-2.9.7-12.el8.x86_64.rpm 422 kB/s | 82 kB 00:00 2021-11-20T05:27:53.195 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:27:53.196 INFO:teuthology.orchestra.run.smithi187.stdout:Total 4.8 MB/s | 967 kB 00:00 2021-11-20T05:27:53.198 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2021-11-20T05:27:53.220 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2021-11-20T05:27:53.220 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2021-11-20T05:27:53.304 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2021-11-20T05:27:53.307 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2021-11-20T05:27:53.486 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2021-11-20T05:27:53.487 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2021-11-20T05:27:53.829 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-11-20T05:27:53.895 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2021-11-20T05:27:54.019 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-20T05:27:54.149 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2021-11-20T05:27:54.357 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-20T05:27:54.404 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-20T05:27:54.405 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2021-11-20T05:27:54.660 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-20T05:27:54.661 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2021-11-20T05:27:54.661 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2021-11-20T05:27:54.661 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2021-11-20T05:27:54.661 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-11-20T05:27:54.662 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-11-20T05:27:54.723 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-11-20T05:27:54.724 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:54.724 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2021-11-20T05:27:54.724 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-11-20T05:27:54.725 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:54.725 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:27:54.888 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librados-devel 2021-11-20T05:27:55.006 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2021-11-20T05:27:55.006 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:55.007 INFO:teuthology.orchestra.run.smithi058.stdout:Installed: 2021-11-20T05:27:55.007 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2021-11-20T05:27:55.007 INFO:teuthology.orchestra.run.smithi058.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2021-11-20T05:27:55.007 INFO:teuthology.orchestra.run.smithi058.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2021-11-20T05:27:55.008 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:55.008 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:27:55.097 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install ceph-fuse 2021-11-20T05:27:55.439 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:55 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:55.666 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:01:55 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:56.784 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:27:56.785 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:56.785 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2021-11-20T05:27:56.786 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:56.786 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2021-11-20T05:27:56.786 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2021-11-20T05:27:56.786 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:56.787 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:27:56.787 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:27:56.787 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2021-11-20T05:27:56.787 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:56.787 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 131 k 2021-11-20T05:27:56.788 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 439 k 2021-11-20T05:27:56.788 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2021-11-20T05:27:56.798 INFO:teuthology.orchestra.run.smithi187.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 10 MB/s | 131 kB 00:00 2021-11-20T05:27:56.799 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:27:56.799 INFO:teuthology.orchestra.run.smithi187.stdout:Total 9.1 MB/s | 131 kB 00:00 2021-11-20T05:27:56.799 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2021-11-20T05:27:56.807 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2021-11-20T05:27:56.807 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2021-11-20T05:27:56.858 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2021-11-20T05:27:56.862 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2021-11-20T05:27:56.976 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2021-11-20T05:27:57.101 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:27:57.103 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:57.103 INFO:teuthology.orchestra.run.smithi058.stdout: Package Architecture Version Repository Size 2021-11-20T05:27:57.103 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:57.103 INFO:teuthology.orchestra.run.smithi058.stdout:Installing: 2021-11-20T05:27:57.104 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2021-11-20T05:27:57.104 INFO:teuthology.orchestra.run.smithi058.stdout:Installing dependencies: 2021-11-20T05:27:57.104 INFO:teuthology.orchestra.run.smithi058.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-11-20T05:27:57.104 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:57.104 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:27:57.105 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:27:57.105 INFO:teuthology.orchestra.run.smithi058.stdout:Install 2 Packages 2021-11-20T05:27:57.105 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:57.105 INFO:teuthology.orchestra.run.smithi058.stdout:Total download size: 967 k 2021-11-20T05:27:57.106 INFO:teuthology.orchestra.run.smithi058.stdout:Installed size: 3.2 M 2021-11-20T05:27:57.106 INFO:teuthology.orchestra.run.smithi058.stdout:Downloading Packages: 2021-11-20T05:27:57.129 INFO:teuthology.orchestra.run.smithi058.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 35 MB/s | 885 kB 00:00 2021-11-20T05:27:57.144 INFO:teuthology.orchestra.run.smithi058.stdout:(2/2): fuse-2.9.7-12.el8.x86_64.rpm 2.0 MB/s | 82 kB 00:00 2021-11-20T05:27:57.145 INFO:teuthology.orchestra.run.smithi058.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:27:57.146 INFO:teuthology.orchestra.run.smithi058.stdout:Total 22 MB/s | 967 kB 00:00 2021-11-20T05:27:57.148 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction check 2021-11-20T05:27:57.172 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction check succeeded. 2021-11-20T05:27:57.173 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction test 2021-11-20T05:27:57.176 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:27:57.263 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction test succeeded. 2021-11-20T05:27:57.267 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction 2021-11-20T05:27:57.461 INFO:teuthology.orchestra.run.smithi058.stdout: Preparing : 1/1 2021-11-20T05:27:57.476 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:27:57.804 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-11-20T05:27:57.888 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:27:57.889 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:57.889 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2021-11-20T05:27:57.890 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2021-11-20T05:27:57.890 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:27:57.891 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:27:58.088 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install libcephfs2 2021-11-20T05:27:58.191 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-20T05:27:58.557 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-20T05:27:58.558 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2021-11-20T05:27:58.645 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:58 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:58.933 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-11-20T05:27:58.933 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:58.934 INFO:teuthology.orchestra.run.smithi058.stdout:Installed: 2021-11-20T05:27:58.934 INFO:teuthology.orchestra.run.smithi058.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-11-20T05:27:58.934 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:27:58.934 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:27:59.135 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install librados-devel 2021-11-20T05:27:59.655 INFO:teuthology.orchestra.run.smithi187.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:27:59.708 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:01:59 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:27:59.949 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:27:59.950 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2021-11-20T05:27:59.950 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:28:00.026 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install libcephfs-devel 2021-11-20T05:28:00.567 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:00 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:01.097 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:28:01.098 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:28:01.098 INFO:teuthology.orchestra.run.smithi058.stdout: Package Architecture Version Repository Size 2021-11-20T05:28:01.098 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:28:01.098 INFO:teuthology.orchestra.run.smithi058.stdout:Installing: 2021-11-20T05:28:01.099 INFO:teuthology.orchestra.run.smithi058.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2021-11-20T05:28:01.099 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:01.099 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:28:01.099 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:28:01.100 INFO:teuthology.orchestra.run.smithi058.stdout:Install 1 Package 2021-11-20T05:28:01.100 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:01.100 INFO:teuthology.orchestra.run.smithi058.stdout:Total download size: 131 k 2021-11-20T05:28:01.100 INFO:teuthology.orchestra.run.smithi058.stdout:Installed size: 439 k 2021-11-20T05:28:01.100 INFO:teuthology.orchestra.run.smithi058.stdout:Downloading Packages: 2021-11-20T05:28:01.110 INFO:teuthology.orchestra.run.smithi058.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 11 MB/s | 131 kB 00:00 2021-11-20T05:28:01.111 INFO:teuthology.orchestra.run.smithi058.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:28:01.112 INFO:teuthology.orchestra.run.smithi058.stdout:Total 9.5 MB/s | 131 kB 00:00 2021-11-20T05:28:01.112 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction check 2021-11-20T05:28:01.119 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction check succeeded. 2021-11-20T05:28:01.119 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction test 2021-11-20T05:28:01.173 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction test succeeded. 2021-11-20T05:28:01.177 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction 2021-11-20T05:28:01.323 INFO:teuthology.orchestra.run.smithi058.stdout: Preparing : 1/1 2021-11-20T05:28:01.556 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:01.829 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:01.870 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:28:01.871 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:28:01.872 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2021-11-20T05:28:01.872 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:28:01.872 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2021-11-20T05:28:01.872 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2021-11-20T05:28:01.873 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:28:01.873 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:28:01.873 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:28:01.873 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2021-11-20T05:28:01.874 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:28:01.874 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 24 k 2021-11-20T05:28:01.874 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 94 k 2021-11-20T05:28:01.874 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2021-11-20T05:28:01.883 INFO:teuthology.orchestra.run.smithi187.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 24 kB 00:00 2021-11-20T05:28:01.883 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:28:01.884 INFO:teuthology.orchestra.run.smithi187.stdout:Total 2.0 MB/s | 24 kB 00:00 2021-11-20T05:28:01.884 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2021-11-20T05:28:01.888 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2021-11-20T05:28:01.888 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2021-11-20T05:28:01.890 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2021-11-20T05:28:01.893 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2021-11-20T05:28:01.950 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2021-11-20T05:28:02.159 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:02.222 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:02.222 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:02.222 INFO:teuthology.orchestra.run.smithi058.stdout:Installed: 2021-11-20T05:28:02.223 INFO:teuthology.orchestra.run.smithi058.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2021-11-20T05:28:02.223 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:02.223 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:28:02.397 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install libcephfs2 2021-11-20T05:28:02.398 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:02.687 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:02.687 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:28:02.688 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2021-11-20T05:28:02.688 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2021-11-20T05:28:02.688 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:28:02.688 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:28:02.771 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librados2 2021-11-20T05:28:02.974 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:02:02 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:03.313 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:03 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:04.061 INFO:teuthology.orchestra.run.smithi058.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:28:04.371 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:28:04.372 INFO:teuthology.orchestra.run.smithi058.stdout:Nothing to do. 2021-11-20T05:28:04.373 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:28:04.444 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install libcephfs-devel 2021-11-20T05:28:05.015 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:02:05 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:05.072 INFO:teuthology.orchestra.run.smithi187.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:28:05.363 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:28:05.364 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2021-11-20T05:28:05.364 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:28:05.871 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librbd1 2021-11-20T05:28:06.406 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:28:06.407 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:28:06.408 INFO:teuthology.orchestra.run.smithi058.stdout: Package Architecture Version Repository Size 2021-11-20T05:28:06.408 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:28:06.408 INFO:teuthology.orchestra.run.smithi058.stdout:Installing: 2021-11-20T05:28:06.408 INFO:teuthology.orchestra.run.smithi058.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2021-11-20T05:28:06.409 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:06.409 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:28:06.409 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:28:06.409 INFO:teuthology.orchestra.run.smithi058.stdout:Install 1 Package 2021-11-20T05:28:06.409 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:06.410 INFO:teuthology.orchestra.run.smithi058.stdout:Total download size: 24 k 2021-11-20T05:28:06.410 INFO:teuthology.orchestra.run.smithi058.stdout:Installed size: 94 k 2021-11-20T05:28:06.410 INFO:teuthology.orchestra.run.smithi058.stdout:Downloading Packages: 2021-11-20T05:28:06.411 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:06 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:06.420 INFO:teuthology.orchestra.run.smithi058.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 24 kB 00:00 2021-11-20T05:28:06.421 INFO:teuthology.orchestra.run.smithi058.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:28:06.421 INFO:teuthology.orchestra.run.smithi058.stdout:Total 1.8 MB/s | 24 kB 00:00 2021-11-20T05:28:06.421 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction check 2021-11-20T05:28:06.425 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction check succeeded. 2021-11-20T05:28:06.426 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction test 2021-11-20T05:28:06.428 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction test succeeded. 2021-11-20T05:28:06.431 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction 2021-11-20T05:28:06.493 INFO:teuthology.orchestra.run.smithi058.stdout: Preparing : 1/1 2021-11-20T05:28:06.704 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:06.953 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:07.271 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:07.271 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:07.271 INFO:teuthology.orchestra.run.smithi058.stdout:Installed: 2021-11-20T05:28:07.272 INFO:teuthology.orchestra.run.smithi058.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2021-11-20T05:28:07.272 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:07.272 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:28:07.382 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install librados2 2021-11-20T05:28:07.395 INFO:teuthology.orchestra.run.smithi187.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:28:07.685 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:28:07.686 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2021-11-20T05:28:07.686 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:28:07.759 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rados 2021-11-20T05:28:07.949 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:02:07 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:08.300 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:08 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:09.219 INFO:teuthology.orchestra.run.smithi058.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:28:09.336 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:28:09.523 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:28:09.524 INFO:teuthology.orchestra.run.smithi058.stdout:Nothing to do. 2021-11-20T05:28:09.525 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:28:09.626 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:28:09.628 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2021-11-20T05:28:09.628 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:28:09.704 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rgw 2021-11-20T05:28:09.996 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install librbd1 2021-11-20T05:28:10.243 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:10 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:10.570 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:02:10 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:11.261 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:28:11.549 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:28:11.550 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2021-11-20T05:28:11.551 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:28:11.620 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-cephfs 2021-11-20T05:28:11.701 INFO:teuthology.orchestra.run.smithi058.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:28:12.009 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:28:12.010 INFO:teuthology.orchestra.run.smithi058.stdout:Nothing to do. 2021-11-20T05:28:12.010 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:28:12.083 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install python3-rados 2021-11-20T05:28:12.160 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:12 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:12.651 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:02:12 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:13.202 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:28:13.495 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:28:13.496 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2021-11-20T05:28:13.496 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:28:13.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rbd 2021-11-20T05:28:13.751 INFO:teuthology.orchestra.run.smithi058.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:28:14.054 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:28:14.056 INFO:teuthology.orchestra.run.smithi058.stdout:Nothing to do. 2021-11-20T05:28:14.056 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:28:14.110 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:14 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:14.124 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install python3-rgw 2021-11-20T05:28:14.692 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:02:14 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:15.152 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:28:15.440 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:28:15.441 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2021-11-20T05:28:15.441 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:28:15.510 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-fuse 2021-11-20T05:28:15.742 INFO:teuthology.orchestra.run.smithi058.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:28:16.045 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:28:16.046 INFO:teuthology.orchestra.run.smithi058.stdout:Nothing to do. 2021-11-20T05:28:16.047 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:28:16.052 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:16 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:16.116 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install python3-cephfs 2021-11-20T05:28:16.686 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:02:16 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:17.367 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:28:17.368 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:28:17.368 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2021-11-20T05:28:17.369 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:28:17.370 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2021-11-20T05:28:17.370 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2021-11-20T05:28:17.371 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:28:17.371 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:28:17.372 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:28:17.372 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2021-11-20T05:28:17.372 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:28:17.373 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 89 k 2021-11-20T05:28:17.373 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 298 k 2021-11-20T05:28:17.374 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2021-11-20T05:28:17.380 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 7.0 MB/s | 89 kB 00:00 2021-11-20T05:28:17.381 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:28:17.381 INFO:teuthology.orchestra.run.smithi187.stdout:Total 6.3 MB/s | 89 kB 00:00 2021-11-20T05:28:17.382 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2021-11-20T05:28:17.393 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2021-11-20T05:28:17.394 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2021-11-20T05:28:17.444 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2021-11-20T05:28:17.447 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2021-11-20T05:28:17.551 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2021-11-20T05:28:17.747 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:17.775 INFO:teuthology.orchestra.run.smithi058.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:28:18.043 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:18.082 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:28:18.083 INFO:teuthology.orchestra.run.smithi058.stdout:Nothing to do. 2021-11-20T05:28:18.083 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:28:18.148 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install python3-rbd 2021-11-20T05:28:18.424 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:18.425 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:28:18.425 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2021-11-20T05:28:18.425 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2021-11-20T05:28:18.425 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:28:18.426 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:28:18.583 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-mirror 2021-11-20T05:28:18.717 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:02:18 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:19.127 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:19 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:19.808 INFO:teuthology.orchestra.run.smithi058.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:28:20.113 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:28:20.114 INFO:teuthology.orchestra.run.smithi058.stdout:Nothing to do. 2021-11-20T05:28:20.115 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:28:20.186 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install rbd-fuse 2021-11-20T05:28:20.436 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:28:20.437 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:28:20.438 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2021-11-20T05:28:20.438 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:28:20.439 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2021-11-20T05:28:20.439 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-20T05:28:20.439 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:28:20.439 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:28:20.439 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:28:20.440 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2021-11-20T05:28:20.440 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:28:20.440 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 3.7 M 2021-11-20T05:28:20.441 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 18 M 2021-11-20T05:28:20.441 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2021-11-20T05:28:20.492 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 69 MB/s | 3.7 MB 00:00 2021-11-20T05:28:20.492 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:28:20.493 INFO:teuthology.orchestra.run.smithi187.stdout:Total 67 MB/s | 3.7 MB 00:00 2021-11-20T05:28:20.493 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2021-11-20T05:28:20.507 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2021-11-20T05:28:20.507 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2021-11-20T05:28:20.572 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2021-11-20T05:28:20.575 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2021-11-20T05:28:20.754 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:02:20 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:21.043 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2021-11-20T05:28:21.273 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:21.666 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:22.111 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:28:22.112 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:28:22.112 INFO:teuthology.orchestra.run.smithi058.stdout: Package Architecture Version Repository Size 2021-11-20T05:28:22.112 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:28:22.113 INFO:teuthology.orchestra.run.smithi058.stdout:Installing: 2021-11-20T05:28:22.113 INFO:teuthology.orchestra.run.smithi058.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2021-11-20T05:28:22.113 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:22.114 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:28:22.114 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:28:22.114 INFO:teuthology.orchestra.run.smithi058.stdout:Install 1 Package 2021-11-20T05:28:22.114 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:22.114 INFO:teuthology.orchestra.run.smithi058.stdout:Total download size: 89 k 2021-11-20T05:28:22.115 INFO:teuthology.orchestra.run.smithi058.stdout:Installed size: 298 k 2021-11-20T05:28:22.115 INFO:teuthology.orchestra.run.smithi058.stdout:Downloading Packages: 2021-11-20T05:28:22.124 INFO:teuthology.orchestra.run.smithi058.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 7.7 MB/s | 89 kB 00:00 2021-11-20T05:28:22.124 INFO:teuthology.orchestra.run.smithi058.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:28:22.125 INFO:teuthology.orchestra.run.smithi058.stdout:Total 6.8 MB/s | 89 kB 00:00 2021-11-20T05:28:22.125 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction check 2021-11-20T05:28:22.138 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction check succeeded. 2021-11-20T05:28:22.139 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction test 2021-11-20T05:28:22.174 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:22.175 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:28:22.175 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2021-11-20T05:28:22.175 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2021-11-20T05:28:22.175 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:28:22.176 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:28:22.192 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction test succeeded. 2021-11-20T05:28:22.195 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction 2021-11-20T05:28:22.306 INFO:teuthology.orchestra.run.smithi058.stdout: Preparing : 1/1 2021-11-20T05:28:22.400 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-nbd 2021-11-20T05:28:22.552 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:22.855 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:22.950 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:22 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:23.228 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:23.229 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:23.229 INFO:teuthology.orchestra.run.smithi058.stdout:Installed: 2021-11-20T05:28:23.229 INFO:teuthology.orchestra.run.smithi058.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2021-11-20T05:28:23.230 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:23.230 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:28:23.423 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install rbd-mirror 2021-11-20T05:28:23.994 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:02:23 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:24.265 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2021-11-20T05:28:24.266 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:28:24.266 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2021-11-20T05:28:24.267 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:28:24.267 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2021-11-20T05:28:24.267 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2021-11-20T05:28:24.267 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:28:24.268 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2021-11-20T05:28:24.268 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2021-11-20T05:28:24.268 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2021-11-20T05:28:24.268 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:28:24.269 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 173 k 2021-11-20T05:28:24.269 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 649 k 2021-11-20T05:28:24.269 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2021-11-20T05:28:24.279 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 13 MB/s | 173 kB 00:00 2021-11-20T05:28:24.280 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:28:24.280 INFO:teuthology.orchestra.run.smithi187.stdout:Total 12 MB/s | 173 kB 00:00 2021-11-20T05:28:24.281 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2021-11-20T05:28:24.293 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2021-11-20T05:28:24.293 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2021-11-20T05:28:24.343 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2021-11-20T05:28:24.347 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2021-11-20T05:28:24.459 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2021-11-20T05:28:24.696 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:24.972 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:25.328 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:25.328 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:28:25.328 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2021-11-20T05:28:25.329 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2021-11-20T05:28:25.329 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:28:25.329 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2021-11-20T05:28:25.437 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:28:25.438 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:28:25.439 INFO:teuthology.orchestra.run.smithi058.stdout: Package Architecture Version Repository Size 2021-11-20T05:28:25.439 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:28:25.439 INFO:teuthology.orchestra.run.smithi058.stdout:Installing: 2021-11-20T05:28:25.440 INFO:teuthology.orchestra.run.smithi058.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-20T05:28:25.440 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:25.440 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:28:25.440 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:28:25.441 INFO:teuthology.orchestra.run.smithi058.stdout:Install 1 Package 2021-11-20T05:28:25.441 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:25.441 INFO:teuthology.orchestra.run.smithi058.stdout:Total download size: 3.7 M 2021-11-20T05:28:25.441 INFO:teuthology.orchestra.run.smithi058.stdout:Installed size: 18 M 2021-11-20T05:28:25.442 INFO:teuthology.orchestra.run.smithi058.stdout:Downloading Packages: 2021-11-20T05:28:25.494 INFO:teuthology.orchestra.run.smithi058.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 68 MB/s | 3.7 MB 00:00 2021-11-20T05:28:25.495 INFO:teuthology.orchestra.run.smithi058.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:28:25.495 INFO:teuthology.orchestra.run.smithi058.stdout:Total 66 MB/s | 3.7 MB 00:00 2021-11-20T05:28:25.496 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction check 2021-11-20T05:28:25.511 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction check succeeded. 2021-11-20T05:28:25.511 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction test 2021-11-20T05:28:25.543 DEBUG:teuthology.parallel:result is None 2021-11-20T05:28:25.580 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction test succeeded. 2021-11-20T05:28:25.584 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction 2021-11-20T05:28:26.098 INFO:teuthology.orchestra.run.smithi058.stdout: Preparing : 1/1 2021-11-20T05:28:26.327 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:26.751 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:27.161 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:27.161 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:27.161 INFO:teuthology.orchestra.run.smithi058.stdout:Installed: 2021-11-20T05:28:27.161 INFO:teuthology.orchestra.run.smithi058.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2021-11-20T05:28:27.162 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:27.162 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:28:27.348 DEBUG:teuthology.orchestra.run.smithi058:> sudo yum -y install rbd-nbd 2021-11-20T05:28:27.919 INFO:teuthology.orchestra.run.smithi058.stdout:Last metadata expiration check: 0:02:27 ago on Sat 20 Nov 2021 05:26:00 AM UTC. 2021-11-20T05:28:29.311 INFO:teuthology.orchestra.run.smithi058.stdout:Dependencies resolved. 2021-11-20T05:28:29.312 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:28:29.313 INFO:teuthology.orchestra.run.smithi058.stdout: Package Architecture Version Repository Size 2021-11-20T05:28:29.313 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:28:29.313 INFO:teuthology.orchestra.run.smithi058.stdout:Installing: 2021-11-20T05:28:29.313 INFO:teuthology.orchestra.run.smithi058.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2021-11-20T05:28:29.314 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:29.314 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction Summary 2021-11-20T05:28:29.314 INFO:teuthology.orchestra.run.smithi058.stdout:================================================================================ 2021-11-20T05:28:29.314 INFO:teuthology.orchestra.run.smithi058.stdout:Install 1 Package 2021-11-20T05:28:29.314 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:29.315 INFO:teuthology.orchestra.run.smithi058.stdout:Total download size: 173 k 2021-11-20T05:28:29.315 INFO:teuthology.orchestra.run.smithi058.stdout:Installed size: 649 k 2021-11-20T05:28:29.315 INFO:teuthology.orchestra.run.smithi058.stdout:Downloading Packages: 2021-11-20T05:28:29.326 INFO:teuthology.orchestra.run.smithi058.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 13 MB/s | 173 kB 00:00 2021-11-20T05:28:29.326 INFO:teuthology.orchestra.run.smithi058.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:28:29.327 INFO:teuthology.orchestra.run.smithi058.stdout:Total 12 MB/s | 173 kB 00:00 2021-11-20T05:28:29.327 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction check 2021-11-20T05:28:29.341 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction check succeeded. 2021-11-20T05:28:29.341 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction test 2021-11-20T05:28:29.395 INFO:teuthology.orchestra.run.smithi058.stdout:Transaction test succeeded. 2021-11-20T05:28:29.399 INFO:teuthology.orchestra.run.smithi058.stdout:Running transaction 2021-11-20T05:28:29.540 INFO:teuthology.orchestra.run.smithi058.stdout: Preparing : 1/1 2021-11-20T05:28:29.792 INFO:teuthology.orchestra.run.smithi058.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:30.063 INFO:teuthology.orchestra.run.smithi058.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:30.436 INFO:teuthology.orchestra.run.smithi058.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:28:30.436 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:30.436 INFO:teuthology.orchestra.run.smithi058.stdout:Installed: 2021-11-20T05:28:30.437 INFO:teuthology.orchestra.run.smithi058.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2021-11-20T05:28:30.437 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:28:30.437 INFO:teuthology.orchestra.run.smithi058.stdout:Complete! 2021-11-20T05:28:30.586 DEBUG:teuthology.parallel:result is None 2021-11-20T05:28:30.587 DEBUG:teuthology.orchestra.run.smithi058:> sudo lsb_release -is 2021-11-20T05:28:30.623 INFO:teuthology.orchestra.run.smithi058.stdout:CentOSStream 2021-11-20T05:28:30.625 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T05:28:30.625 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T05:28:30.626 INFO:teuthology.packaging:ref: None 2021-11-20T05:28:30.626 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T05:28:30.627 INFO:teuthology.packaging:branch: None 2021-11-20T05:28:30.627 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T05:28:30.843 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T05:28:30.845 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T05:28:31.084 DEBUG:teuthology.orchestra.run.smithi058:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-11-20T05:28:31.107 INFO:teuthology.orchestra.run.smithi058.stdout:16.2.4-0.el8 2021-11-20T05:28:31.108 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2021-11-20T05:28:31.108 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2021-11-20T05:28:31.109 DEBUG:teuthology.orchestra.run.smithi187:> sudo lsb_release -is 2021-11-20T05:28:31.150 INFO:teuthology.orchestra.run.smithi187.stdout:CentOSStream 2021-11-20T05:28:31.152 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T05:28:31.152 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T05:28:31.153 INFO:teuthology.packaging:ref: None 2021-11-20T05:28:31.153 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T05:28:31.154 INFO:teuthology.packaging:branch: None 2021-11-20T05:28:31.154 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T05:28:31.362 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T05:28:31.363 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T05:28:31.571 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-11-20T05:28:31.596 INFO:teuthology.orchestra.run.smithi187.stdout:16.2.4-0.el8 2021-11-20T05:28:31.596 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2021-11-20T05:28:31.596 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2021-11-20T05:28:31.597 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-11-20T05:28:31.597 DEBUG:teuthology.orchestra.run.smithi058:> set -ex 2021-11-20T05:28:31.598 DEBUG:teuthology.orchestra.run.smithi058:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-11-20T05:28:31.630 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2021-11-20T05:28:31.631 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-11-20T05:28:31.664 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-11-20T05:28:31.665 DEBUG:teuthology.orchestra.run.smithi058:> set -ex 2021-11-20T05:28:31.666 DEBUG:teuthology.orchestra.run.smithi058:> sudo dd of=/usr/bin/daemon-helper 2021-11-20T05:28:31.701 DEBUG:teuthology.orchestra.run.smithi058:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-11-20T05:28:31.771 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2021-11-20T05:28:31.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/daemon-helper 2021-11-20T05:28:31.801 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-11-20T05:28:31.868 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-11-20T05:28:31.868 DEBUG:teuthology.orchestra.run.smithi058:> set -ex 2021-11-20T05:28:31.869 DEBUG:teuthology.orchestra.run.smithi058:> sudo dd of=/usr/bin/adjust-ulimits 2021-11-20T05:28:31.900 DEBUG:teuthology.orchestra.run.smithi058:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-11-20T05:28:31.970 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2021-11-20T05:28:31.971 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/adjust-ulimits 2021-11-20T05:28:31.997 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-11-20T05:28:32.064 INFO:teuthology.run_tasks:Running task cephadm... 2021-11-20T05:28:32.139 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa'} 2021-11-20T05:28:32.140 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2021-11-20T05:28:32.141 INFO:tasks.cephadm:Cluster fsid is b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:28:32.141 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2021-11-20T05:28:32.142 INFO:tasks.cephadm:No mon roles; fabricating mons 2021-11-20T05:28:32.142 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi058': '172.21.15.58', 'mon.smithi187': '172.21.15.187'} 2021-11-20T05:28:32.142 INFO:tasks.cephadm:Normalizing hostnames... 2021-11-20T05:28:32.143 DEBUG:teuthology.orchestra.run.smithi058:> sudo hostname $(hostname -s) 2021-11-20T05:28:32.175 DEBUG:teuthology.orchestra.run.smithi187:> sudo hostname $(hostname -s) 2021-11-20T05:28:32.201 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2021-11-20T05:28:32.202 DEBUG:teuthology.orchestra.run.smithi058:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2021-11-20T05:28:32.505 INFO:teuthology.orchestra.run.smithi058.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 20 05:28 /home/ubuntu/cephtest/cephadm 2021-11-20T05:28:32.505 DEBUG:teuthology.orchestra.run.smithi187:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2021-11-20T05:28:32.670 INFO:teuthology.orchestra.run.smithi187.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 20 05:28 /home/ubuntu/cephtest/cephadm 2021-11-20T05:28:32.670 DEBUG:teuthology.orchestra.run.smithi058:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-11-20T05:28:32.691 DEBUG:teuthology.orchestra.run.smithi187:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-11-20T05:28:32.721 DEBUG:teuthology.orchestra.run.smithi058:> sudo mkdir -p /etc/ceph 2021-11-20T05:28:32.762 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph 2021-11-20T05:28:32.789 DEBUG:teuthology.orchestra.run.smithi058:> sudo chmod 777 /etc/ceph 2021-11-20T05:28:32.832 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 777 /etc/ceph 2021-11-20T05:28:32.860 INFO:tasks.cephadm:Writing seed config... 2021-11-20T05:28:32.861 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2021-11-20T05:28:32.862 INFO:tasks.cephadm: override: [client] debug client = 20 2021-11-20T05:28:32.862 INFO:tasks.cephadm: override: [client] debug ms = 1 2021-11-20T05:28:32.862 INFO:tasks.cephadm: override: [client] rados mon op timeout = 15m 2021-11-20T05:28:32.863 INFO:tasks.cephadm: override: [client] rados osd op timeout = 15m 2021-11-20T05:28:32.863 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2021-11-20T05:28:32.863 INFO:tasks.cephadm: override: [mds] debug mds = 20 2021-11-20T05:28:32.864 INFO:tasks.cephadm: override: [mds] debug ms = 1 2021-11-20T05:28:32.864 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2021-11-20T05:28:32.864 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2021-11-20T05:28:32.865 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2021-11-20T05:28:32.865 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2021-11-20T05:28:32.865 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2021-11-20T05:28:32.865 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 15m 2021-11-20T05:28:32.866 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 15m 2021-11-20T05:28:32.866 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2021-11-20T05:28:32.866 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2021-11-20T05:28:32.867 INFO:tasks.cephadm: override: [mon] debug mon = 20 2021-11-20T05:28:32.867 INFO:tasks.cephadm: override: [mon] debug ms = 1 2021-11-20T05:28:32.867 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2021-11-20T05:28:32.868 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2021-11-20T05:28:32.868 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2021-11-20T05:28:32.868 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2021-11-20T05:28:32.869 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2021-11-20T05:28:32.869 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2021-11-20T05:28:32.869 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2021-11-20T05:28:32.870 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2021-11-20T05:28:32.870 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2021-11-20T05:28:32.870 INFO:tasks.cephadm: override: [osd] debug ms = 1 2021-11-20T05:28:32.871 INFO:tasks.cephadm: override: [osd] debug osd = 20 2021-11-20T05:28:32.871 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2021-11-20T05:28:32.871 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2021-11-20T05:28:32.872 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2021-11-20T05:28:32.872 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2021-11-20T05:28:32.872 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2021-11-20T05:28:32.872 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2021-11-20T05:28:32.873 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2021-11-20T05:28:32.874 DEBUG:teuthology.orchestra.run.smithi058:> set -ex 2021-11-20T05:28:32.874 DEBUG:teuthology.orchestra.run.smithi058:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2021-11-20T05:28:32.892 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = b33f8c4a-49c2-11ec-8c2c-001a4aab830c mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 15m rados osd op timeout = 15m [mds] debug mds = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 15m rados osd op timeout = 15m 2021-11-20T05:28:32.893 DEBUG:teuthology.orchestra.run.smithi058:mon.smithi058> sudo journalctl -f -n 0 -u ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mon.smithi058.service 2021-11-20T05:28:32.935 INFO:tasks.cephadm:Bootstrapping... 2021-11-20T05:28:32.935 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.58 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2021-11-20T05:28:32.962 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:-- Logs begin at Sat 2021-11-20 05:17:50 UTC. -- 2021-11-20T05:28:33.217 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-11-20T05:28:34.075 INFO:teuthology.orchestra.run.smithi058.stderr:/bin/podman: 3.4.1-dev 2021-11-20T05:28:34.145 INFO:teuthology.orchestra.run.smithi058.stderr:Verifying podman|docker is present... 2021-11-20T05:28:34.146 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-11-20T05:28:34.189 INFO:teuthology.orchestra.run.smithi058.stderr:/bin/podman: 3.4.1-dev 2021-11-20T05:28:34.293 INFO:teuthology.orchestra.run.smithi058.stderr:Verifying lvm2 is present... 2021-11-20T05:28:34.293 INFO:teuthology.orchestra.run.smithi058.stderr:Verifying time synchronization is in place... 2021-11-20T05:28:34.293 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl is-enabled chrony.service 2021-11-20T05:28:34.300 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-11-20T05:28:34.301 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl is-active chrony.service 2021-11-20T05:28:34.307 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: inactive 2021-11-20T05:28:34.307 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl is-enabled chronyd.service 2021-11-20T05:28:34.313 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: enabled 2021-11-20T05:28:34.314 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl is-active chronyd.service 2021-11-20T05:28:34.319 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: active 2021-11-20T05:28:34.320 INFO:teuthology.orchestra.run.smithi058.stderr:Unit chronyd.service is enabled and running 2021-11-20T05:28:34.321 INFO:teuthology.orchestra.run.smithi058.stderr:Repeating the final host check... 2021-11-20T05:28:34.321 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-11-20T05:28:34.360 INFO:teuthology.orchestra.run.smithi058.stderr:/bin/podman: 3.4.1-dev 2021-11-20T05:28:34.472 INFO:teuthology.orchestra.run.smithi058.stderr:podman|docker (/bin/podman) is present 2021-11-20T05:28:34.472 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl is present 2021-11-20T05:28:34.473 INFO:teuthology.orchestra.run.smithi058.stderr:lvcreate is present 2021-11-20T05:28:34.473 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl is-enabled chrony.service 2021-11-20T05:28:34.478 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-11-20T05:28:34.478 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl is-active chrony.service 2021-11-20T05:28:34.483 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: inactive 2021-11-20T05:28:34.484 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl is-enabled chronyd.service 2021-11-20T05:28:34.490 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: enabled 2021-11-20T05:28:34.491 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl is-active chronyd.service 2021-11-20T05:28:34.496 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: active 2021-11-20T05:28:34.497 INFO:teuthology.orchestra.run.smithi058.stderr:Unit chronyd.service is enabled and running 2021-11-20T05:28:34.497 INFO:teuthology.orchestra.run.smithi058.stderr:Host looks OK 2021-11-20T05:28:34.497 INFO:teuthology.orchestra.run.smithi058.stderr:Cluster fsid: b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:28:34.498 INFO:teuthology.orchestra.run.smithi058.stderr:Acquiring lock 140284633807560 on /run/cephadm/b33f8c4a-49c2-11ec-8c2c-001a4aab830c.lock 2021-11-20T05:28:34.498 INFO:teuthology.orchestra.run.smithi058.stderr:Lock 140284633807560 acquired on /run/cephadm/b33f8c4a-49c2-11ec-8c2c-001a4aab830c.lock 2021-11-20T05:28:34.499 INFO:teuthology.orchestra.run.smithi058.stderr:Verifying IP 172.21.15.58 port 3300 ... 2021-11-20T05:28:34.499 INFO:teuthology.orchestra.run.smithi058.stderr:Verifying IP 172.21.15.58 port 6789 ... 2021-11-20T05:28:34.499 INFO:teuthology.orchestra.run.smithi058.stderr:Base mon IP is 172.21.15.58, final addrv is [v2:172.21.15.58:3300,v1:172.21.15.58:6789] 2021-11-20T05:28:34.500 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /sbin/ip route ls 2021-11-20T05:28:34.500 INFO:teuthology.orchestra.run.smithi058.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2021-11-20T05:28:34.500 INFO:teuthology.orchestra.run.smithi058.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.58 metric 100 2021-11-20T05:28:34.501 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /sbin/ip -6 route ls 2021-11-20T05:28:34.503 INFO:teuthology.orchestra.run.smithi058.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2021-11-20T05:28:34.503 INFO:teuthology.orchestra.run.smithi058.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2021-11-20T05:28:34.504 INFO:teuthology.orchestra.run.smithi058.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1796sec hoplimit 64 pref medium 2021-11-20T05:28:34.504 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /sbin/ip -6 addr ls 2021-11-20T05:28:34.506 INFO:teuthology.orchestra.run.smithi058.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2021-11-20T05:28:34.506 INFO:teuthology.orchestra.run.smithi058.stderr:/sbin/ip: inet6 ::1/128 scope host 2021-11-20T05:28:34.507 INFO:teuthology.orchestra.run.smithi058.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-11-20T05:28:34.507 INFO:teuthology.orchestra.run.smithi058.stderr:/sbin/ip: 3: ens1f0: mtu 1500 state UP qlen 1000 2021-11-20T05:28:34.507 INFO:teuthology.orchestra.run.smithi058.stderr:/sbin/ip: inet6 fe80::ae1f:6bff:fea5:c8e4/64 scope link noprefixroute 2021-11-20T05:28:34.507 INFO:teuthology.orchestra.run.smithi058.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-11-20T05:28:34.508 INFO:teuthology.orchestra.run.smithi058.stderr:Mon IP 172.21.15.58 is in CIDR network 172.21.0.0/20 2021-11-20T05:28:34.509 INFO:teuthology.orchestra.run.smithi058.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2021-11-20T05:28:34.509 INFO:teuthology.orchestra.run.smithi058.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2021-11-20T05:28:34.509 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2021-11-20T05:28:34.578 INFO:teuthology.orchestra.run.smithi058.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2021-11-20T05:28:34.879 INFO:teuthology.orchestra.run.smithi058.stderr:/bin/podman: Getting image source signatures 2021-11-20T05:28:34.903 INFO:teuthology.orchestra.run.smithi058.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2021-11-20T05:28:34.903 INFO:teuthology.orchestra.run.smithi058.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2021-11-20T05:28:35.024 INFO:teuthology.orchestra.run.smithi058.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2021-11-20T05:28:35.074 INFO:teuthology.orchestra.run.smithi058.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2021-11-20T05:28:47.539 INFO:teuthology.orchestra.run.smithi058.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2021-11-20T05:28:47.605 INFO:teuthology.orchestra.run.smithi058.stderr:/bin/podman: Writing manifest to image destination 2021-11-20T05:28:47.605 INFO:teuthology.orchestra.run.smithi058.stderr:/bin/podman: Storing signatures 2021-11-20T05:28:48.052 INFO:teuthology.orchestra.run.smithi058.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2021-11-20T05:28:48.079 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2021-11-20T05:28:48.719 INFO:teuthology.orchestra.run.smithi058.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2021-11-20T05:28:49.124 INFO:teuthology.orchestra.run.smithi058.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2021-11-20T05:28:49.124 INFO:teuthology.orchestra.run.smithi058.stderr:Extracting ceph user uid/gid from container image... 2021-11-20T05:28:49.124 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2021-11-20T05:28:49.530 INFO:teuthology.orchestra.run.smithi058.stderr:stat: 167 167 2021-11-20T05:28:49.971 INFO:teuthology.orchestra.run.smithi058.stderr:Creating initial keys... 2021-11-20T05:28:49.972 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2021-11-20T05:28:50.405 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-authtool: AQCSh5hhNmCRFRAAXE8hJj6YzGect7J6AiPblA== 2021-11-20T05:28:50.726 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2021-11-20T05:28:51.164 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-authtool: AQCTh5hhIxwbCRAAUe0kQTUl/j/2jwQ6p9uYxA== 2021-11-20T05:28:51.468 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2021-11-20T05:28:51.880 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-authtool: AQCTh5hhZkGjMxAAKvQpip9l1AzkuXUJsOchoA== 2021-11-20T05:28:52.210 INFO:teuthology.orchestra.run.smithi058.stderr:Creating initial monmap... 2021-11-20T05:28:52.211 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpjj1o0v1c:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c --addv smithi058 [v2:172.21.15.58:3300,v1:172.21.15.58:6789] /tmp/monmap 2021-11-20T05:28:52.622 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2021-11-20T05:28:52.622 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:28:52.623 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-11-20T05:28:52.954 INFO:teuthology.orchestra.run.smithi058.stderr:monmaptool for smithi058 [v2:172.21.15.58:3300,v1:172.21.15.58:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2021-11-20T05:28:52.954 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/monmaptool: set fsid to b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:28:52.954 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-11-20T05:28:52.954 INFO:teuthology.orchestra.run.smithi058.stderr: 2021-11-20T05:28:52.955 INFO:teuthology.orchestra.run.smithi058.stderr:Creating mon... 2021-11-20T05:28:52.955 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058:/var/lib/ceph/mon/ceph-smithi058:z -v /tmp/ceph-tmpow51dckh:/tmp/keyring:z -v /tmp/ceph-tmpjj1o0v1c:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi058 --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2021-11-20T05:28:53.399 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.368+0000 7f662b79a700 0 set uid:gid to 167:167 (ceph:ceph) 2021-11-20T05:28:53.399 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.368+0000 7f662b79a700 1 imported monmap: 2021-11-20T05:28:53.399 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: epoch 0 2021-11-20T05:28:53.400 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:28:53.400 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: last_changed 2021-11-20T05:28:52.613926+0000 2021-11-20T05:28:53.400 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: created 2021-11-20T05:28:52.613926+0000 2021-11-20T05:28:53.400 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2021-11-20T05:28:53.401 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: election_strategy: 1 2021-11-20T05:28:53.401 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] mon.smithi058 2021-11-20T05:28:53.401 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.401 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.368+0000 7f662b79a700 0 /usr/bin/ceph-mon: set fsid to b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:28:53.402 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: RocksDB version: 6.8.1 2021-11-20T05:28:53.402 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.402 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-11-20T05:28:53.402 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Compile date May 13 2021 2021-11-20T05:28:53.402 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: DB SUMMARY 2021-11-20T05:28:53.403 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.403 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi058/store.db dir, Total Num: 0, files: 2021-11-20T05:28:53.403 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.403 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi058/store.db: 2021-11-20T05:28:53.404 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.404 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.error_if_exists: 0 2021-11-20T05:28:53.404 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.create_if_missing: 1 2021-11-20T05:28:53.404 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.paranoid_checks: 1 2021-11-20T05:28:53.405 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.env: 0x555b63ba11c0 2021-11-20T05:28:53.405 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.fs: Posix File System 2021-11-20T05:28:53.405 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.info_log: 0x555b65846ec0 2021-11-20T05:28:53.405 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.max_file_opening_threads: 16 2021-11-20T05:28:53.405 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.statistics: (nil) 2021-11-20T05:28:53.406 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.use_fsync: 0 2021-11-20T05:28:53.406 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.max_log_file_size: 0 2021-11-20T05:28:53.406 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-11-20T05:28:53.406 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-11-20T05:28:53.407 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.keep_log_file_num: 1000 2021-11-20T05:28:53.407 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.recycle_log_file_num: 0 2021-11-20T05:28:53.407 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.allow_fallocate: 1 2021-11-20T05:28:53.407 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.allow_mmap_reads: 0 2021-11-20T05:28:53.407 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.allow_mmap_writes: 0 2021-11-20T05:28:53.408 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.use_direct_reads: 0 2021-11-20T05:28:53.408 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-11-20T05:28:53.408 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.create_missing_column_families: 0 2021-11-20T05:28:53.408 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.db_log_dir: 2021-11-20T05:28:53.409 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi058/store.db 2021-11-20T05:28:53.409 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-11-20T05:28:53.409 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.max_subcompactions: 1 2021-11-20T05:28:53.409 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.max_background_flushes: -1 2021-11-20T05:28:53.409 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-11-20T05:28:53.410 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-11-20T05:28:53.410 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-11-20T05:28:53.410 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-11-20T05:28:53.410 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-11-20T05:28:53.411 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.advise_random_on_open: 1 2021-11-20T05:28:53.411 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.db_write_buffer_size: 0 2021-11-20T05:28:53.411 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.write_buffer_manager: 0x555b6584d530 2021-11-20T05:28:53.411 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-11-20T05:28:53.411 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-11-20T05:28:53.412 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-11-20T05:28:53.412 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-11-20T05:28:53.412 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.rate_limiter: (nil) 2021-11-20T05:28:53.412 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-11-20T05:28:53.412 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.wal_recovery_mode: 2 2021-11-20T05:28:53.413 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.enable_thread_tracking: 0 2021-11-20T05:28:53.413 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.enable_pipelined_write: 0 2021-11-20T05:28:53.413 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.unordered_write: 0 2021-11-20T05:28:53.413 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-11-20T05:28:53.414 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-11-20T05:28:53.414 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-11-20T05:28:53.414 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-11-20T05:28:53.414 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.row_cache: None 2021-11-20T05:28:53.415 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.wal_filter: None 2021-11-20T05:28:53.415 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-11-20T05:28:53.415 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.allow_ingest_behind: 0 2021-11-20T05:28:53.415 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.preserve_deletes: 0 2021-11-20T05:28:53.415 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.two_write_queues: 0 2021-11-20T05:28:53.416 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.manual_wal_flush: 0 2021-11-20T05:28:53.416 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.atomic_flush: 0 2021-11-20T05:28:53.416 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-11-20T05:28:53.416 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-11-20T05:28:53.417 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-11-20T05:28:53.417 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.log_readahead_size: 0 2021-11-20T05:28:53.417 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-11-20T05:28:53.417 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.max_background_jobs: 2 2021-11-20T05:28:53.417 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.max_background_compactions: -1 2021-11-20T05:28:53.418 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-11-20T05:28:53.418 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-11-20T05:28:53.418 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-11-20T05:28:53.418 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.max_total_wal_size: 0 2021-11-20T05:28:53.419 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-11-20T05:28:53.419 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-11-20T05:28:53.419 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-11-20T05:28:53.419 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-11-20T05:28:53.419 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.max_open_files: -1 2021-11-20T05:28:53.420 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.bytes_per_sync: 0 2021-11-20T05:28:53.420 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-11-20T05:28:53.420 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-11-20T05:28:53.420 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Options.compaction_readahead_size: 0 2021-11-20T05:28:53.421 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Compression algorithms supported: 2021-11-20T05:28:53.421 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-11-20T05:28:53.421 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: kZSTD supported: 0 2021-11-20T05:28:53.421 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: kXpressCompression supported: 0 2021-11-20T05:28:53.421 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: kLZ4HCCompression supported: 1 2021-11-20T05:28:53.422 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: kLZ4Compression supported: 1 2021-11-20T05:28:53.422 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: kBZip2Compression supported: 0 2021-11-20T05:28:53.422 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: kZlibCompression supported: 1 2021-11-20T05:28:53.422 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: kSnappyCompression supported: 1 2021-11-20T05:28:53.423 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-11-20T05:28:53.423 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.369+0000 7f662b79a700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2021-11-20T05:28:53.423 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.423 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi058/store.db/MANIFEST-000001 2021-11-20T05:28:53.423 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.424 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-11-20T05:28:53.424 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.424 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-11-20T05:28:53.424 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.merge_operator: 2021-11-20T05:28:53.424 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compaction_filter: None 2021-11-20T05:28:53.425 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compaction_filter_factory: None 2021-11-20T05:28:53.425 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-11-20T05:28:53.425 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-11-20T05:28:53.425 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555b6575e120) 2021-11-20T05:28:53.426 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2021-11-20T05:28:53.426 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2021-11-20T05:28:53.426 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2021-11-20T05:28:53.426 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2021-11-20T05:28:53.426 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: index_type: 0 2021-11-20T05:28:53.427 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2021-11-20T05:28:53.427 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: index_shortening: 1 2021-11-20T05:28:53.427 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2021-11-20T05:28:53.427 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2021-11-20T05:28:53.428 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: checksum: 1 2021-11-20T05:28:53.428 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2021-11-20T05:28:53.428 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: block_cache: 0x555b65794d10 2021-11-20T05:28:53.428 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2021-11-20T05:28:53.428 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: block_cache_options: 2021-11-20T05:28:53.429 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: capacity : 536870912 2021-11-20T05:28:53.429 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2021-11-20T05:28:53.429 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2021-11-20T05:28:53.429 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2021-11-20T05:28:53.429 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2021-11-20T05:28:53.430 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2021-11-20T05:28:53.430 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: block_size: 4096 2021-11-20T05:28:53.430 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2021-11-20T05:28:53.430 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2021-11-20T05:28:53.431 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2021-11-20T05:28:53.431 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2021-11-20T05:28:53.431 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: partition_filters: 0 2021-11-20T05:28:53.431 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2021-11-20T05:28:53.431 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2021-11-20T05:28:53.432 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2021-11-20T05:28:53.432 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: verify_compression: 0 2021-11-20T05:28:53.432 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2021-11-20T05:28:53.432 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: format_version: 2 2021-11-20T05:28:53.432 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2021-11-20T05:28:53.433 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: block_align: 0 2021-11-20T05:28:53.433 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.433 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.write_buffer_size: 33554432 2021-11-20T05:28:53.433 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.max_write_buffer_number: 2 2021-11-20T05:28:53.434 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compression: NoCompression 2021-11-20T05:28:53.434 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.bottommost_compression: Disabled 2021-11-20T05:28:53.434 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.prefix_extractor: nullptr 2021-11-20T05:28:53.434 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-11-20T05:28:53.434 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.num_levels: 7 2021-11-20T05:28:53.435 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-11-20T05:28:53.435 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-11-20T05:28:53.435 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-11-20T05:28:53.435 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-11-20T05:28:53.436 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-11-20T05:28:53.436 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-11-20T05:28:53.436 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-11-20T05:28:53.436 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-11-20T05:28:53.436 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-11-20T05:28:53.437 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-11-20T05:28:53.437 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compression_opts.level: 32767 2021-11-20T05:28:53.437 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compression_opts.strategy: 0 2021-11-20T05:28:53.437 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-11-20T05:28:53.438 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-11-20T05:28:53.438 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compression_opts.enabled: false 2021-11-20T05:28:53.438 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-11-20T05:28:53.438 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-11-20T05:28:53.438 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-11-20T05:28:53.439 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.target_file_size_base: 67108864 2021-11-20T05:28:53.439 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-11-20T05:28:53.439 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-11-20T05:28:53.439 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-11-20T05:28:53.440 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-11-20T05:28:53.440 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-11-20T05:28:53.440 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-11-20T05:28:53.440 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-11-20T05:28:53.440 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-11-20T05:28:53.441 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-11-20T05:28:53.441 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-11-20T05:28:53.441 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-11-20T05:28:53.441 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-11-20T05:28:53.441 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-11-20T05:28:53.442 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.arena_block_size: 4194304 2021-11-20T05:28:53.442 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-11-20T05:28:53.442 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-11-20T05:28:53.442 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-11-20T05:28:53.443 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.disable_auto_compactions: 0 2021-11-20T05:28:53.443 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-11-20T05:28:53.443 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-11-20T05:28:53.443 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-11-20T05:28:53.443 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-11-20T05:28:53.444 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-11-20T05:28:53.444 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-11-20T05:28:53.444 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-11-20T05:28:53.444 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-11-20T05:28:53.445 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-11-20T05:28:53.445 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-11-20T05:28:53.445 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.table_properties_collectors: 2021-11-20T05:28:53.445 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.inplace_update_support: 0 2021-11-20T05:28:53.445 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-11-20T05:28:53.446 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-11-20T05:28:53.446 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-11-20T05:28:53.446 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-11-20T05:28:53.446 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.bloom_locality: 0 2021-11-20T05:28:53.446 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.max_successive_merges: 0 2021-11-20T05:28:53.447 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-11-20T05:28:53.447 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.paranoid_file_checks: 0 2021-11-20T05:28:53.447 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.force_consistency_checks: 0 2021-11-20T05:28:53.447 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.report_bg_io_stats: 0 2021-11-20T05:28:53.448 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.ttl: 2592000 2021-11-20T05:28:53.448 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-11-20T05:28:53.448 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi058/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-11-20T05:28:53.448 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.448 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.370+0000 7f662b79a700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-11-20T05:28:53.449 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.449 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.371+0000 7f662b79a700 4 rocksdb: DB pointer 0x555b65859800 2021-11-20T05:28:53.449 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.371+0000 7f6614633700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-11-20T05:28:53.449 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.371+0000 7f6614633700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-11-20T05:28:53.450 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2021-11-20T05:28:53.450 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T05:28:53.450 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-11-20T05:28:53.450 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-11-20T05:28:53.451 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T05:28:53.451 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-11-20T05:28:53.451 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-11-20T05:28:53.451 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T05:28:53.451 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.452 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-20T05:28:53.452 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:28:53.452 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:28:53.452 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T05:28:53.452 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T05:28:53.453 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.453 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-20T05:28:53.453 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:28:53.453 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:28:53.454 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T05:28:53.454 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T05:28:53.454 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T05:28:53.454 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T05:28:53.454 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T05:28:53.455 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-11-20T05:28:53.455 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:28:53.455 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:28:53.455 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T05:28:53.456 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.456 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-11-20T05:28:53.456 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.456 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-20T05:28:53.456 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:28:53.457 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:28:53.457 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T05:28:53.457 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T05:28:53.457 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.458 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-20T05:28:53.458 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:28:53.458 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:28:53.458 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T05:28:53.458 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T05:28:53.459 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T05:28:53.459 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T05:28:53.459 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T05:28:53.459 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-11-20T05:28:53.459 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:28:53.460 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:28:53.460 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T05:28:53.460 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.460 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-11-20T05:28:53.461 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: 2021-11-20T05:28:53.461 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.373+0000 7f662b79a700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2021-11-20T05:28:53.461 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.373+0000 7f662b79a700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2021-11-20T05:28:53.461 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:28:53.373+0000 7f662b79a700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi058 for mon.smithi058 2021-11-20T05:28:53.798 INFO:teuthology.orchestra.run.smithi058.stderr:create mon.smithi058 on 2021-11-20T05:28:53.798 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:28:53.804 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl disable ceph.target 2021-11-20T05:28:53.811 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2021-11-20T05:28:53.897 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl enable ceph.target 2021-11-20T05:28:53.903 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2021-11-20T05:28:53.986 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl start ceph.target 2021-11-20T05:28:53.992 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl enable ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c.target 2021-11-20T05:28:53.998 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c.target → /etc/systemd/system/ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c.target. 2021-11-20T05:28:53.998 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c.target → /etc/systemd/system/ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c.target. 2021-11-20T05:28:54.080 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl start ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c.target 2021-11-20T05:28:54.087 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl daemon-reload 2021-11-20T05:28:54.174 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl stop ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mon.smithi058 2021-11-20T05:28:54.206 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl reset-failed ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mon.smithi058 2021-11-20T05:28:54.211 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: Failed to reset failed state of unit ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mon.smithi058.service: Unit ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mon.smithi058.service not loaded. 2021-11-20T05:28:54.212 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl enable ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mon.smithi058 2021-11-20T05:28:54.219 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: Created symlink /etc/systemd/system/ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c.target.wants/ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mon.smithi058.service → /etc/systemd/system/ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@.service. 2021-11-20T05:28:54.304 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl start ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mon.smithi058 2021-11-20T05:28:54.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:54 smithi058 systemd[1]: Starting Ceph mon.smithi058 for b33f8c4a-49c2-11ec-8c2c-001a4aab830c... 2021-11-20T05:28:54.835 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:54 smithi058 podman[32804]: 2021-11-20T05:28:54.835 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:54 smithi058 bash[32730]: 484820f376fb75acb54c28c8951138f20fcd3bc0723603545f41d7fc50267ead 2021-11-20T05:28:54.835 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:54 smithi058 conmon[32845]: cluster 2021-11-20T05:28:54.787463+0000 mon.smithi058 (mon.0) 0 : [INF] mkfs b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:28:54.836 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:54 smithi058 conmon[32845]: cluster 2021-11-20T05 2021-11-20T05:28:54.836 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:54 smithi058 conmon[32845]: :28:54.784401+0000 mon.smithi058 (mon.0) 1 : cluster [INF] mon.smithi058 is new leader, mons smithi058 in quorum (ranks 0) 2021-11-20T05:28:54.837 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-20T05:28:54.846 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: disabled 2021-11-20T05:28:54.847 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl is-active firewalld.service 2021-11-20T05:28:54.853 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: inactive 2021-11-20T05:28:54.853 INFO:teuthology.orchestra.run.smithi058.stderr:firewalld.service is not enabled 2021-11-20T05:28:54.854 INFO:teuthology.orchestra.run.smithi058.stderr:Not possible to enable service . firewalld.service is not available 2021-11-20T05:28:54.854 INFO:teuthology.orchestra.run.smithi058.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-11-20T05:28:54.855 INFO:teuthology.orchestra.run.smithi058.stderr:Waiting for mon to start... 2021-11-20T05:28:54.855 INFO:teuthology.orchestra.run.smithi058.stderr:Waiting for mon... 2021-11-20T05:28:54.855 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058:/var/lib/ceph/mon/ceph-smithi058:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2021-11-20T05:28:55.100 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:54 smithi058 systemd[1]: Started Ceph mon.smithi058 for b33f8c4a-49c2-11ec-8c2c-001a4aab830c. 2021-11-20T05:28:55.333 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.328+0000 7f2048a16700 1 Processor -- start 2021-11-20T05:28:55.333 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.328+0000 7f2048a16700 1 -- start start 2021-11-20T05:28:55.334 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.328+0000 7f2048a16700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f20440fc000 0x7f20440fc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:28:55.334 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.328+0000 7f2048a16700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20440fc910 con 0x7f20440fc000 2021-11-20T05:28:55.334 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.329+0000 7f20437fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f20440fc000 0x7f20440fc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:28:55.335 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.329+0000 7f20437fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f20440fc000 0x7f20440fc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39494/0 (socket says 172.21.15.58:39494) 2021-11-20T05:28:55.335 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.329+0000 7f20437fe700 1 -- 172.21.15.58:0/4049157387 learned_addr learned my addr 172.21.15.58:0/4049157387 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:28:55.335 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.329+0000 7f20437fe700 1 -- 172.21.15.58:0/4049157387 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20440fca50 con 0x7f20440fc000 2021-11-20T05:28:55.336 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.329+0000 7f20437fe700 1 --2- 172.21.15.58:0/4049157387 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f20440fc000 0x7f20440fc420 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f203400b5f0 tx=0x7f2034006ee0).ready entity=mon.0 client_cookie=72dd9fc71cd4c297 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:28:55.336 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.329+0000 7f20427fc700 1 -- 172.21.15.58:0/4049157387 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f203400d600 con 0x7f20440fc000 2021-11-20T05:28:55.337 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.329+0000 7f20427fc700 1 -- 172.21.15.58:0/4049157387 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203400d760 con 0x7f20440fc000 2021-11-20T05:28:55.337 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.329+0000 7f20427fc700 1 -- 172.21.15.58:0/4049157387 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f203400da30 con 0x7f20440fc000 2021-11-20T05:28:55.337 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.330+0000 7f2048a16700 1 -- 172.21.15.58:0/4049157387 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f20440fc000 msgr2=0x7f20440fc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:28:55.338 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.330+0000 7f2048a16700 1 --2- 172.21.15.58:0/4049157387 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f20440fc000 0x7f20440fc420 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f203400b5f0 tx=0x7f2034006ee0).stop 2021-11-20T05:28:55.338 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.330+0000 7f2048a16700 1 -- 172.21.15.58:0/4049157387 shutdown_connections 2021-11-20T05:28:55.338 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.330+0000 7f2048a16700 1 --2- 172.21.15.58:0/4049157387 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f20440fc000 0x7f20440fc420 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:28:55.339 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.330+0000 7f2048a16700 1 -- 172.21.15.58:0/4049157387 >> 172.21.15.58:0/4049157387 conn(0x7f20440f5870 msgr2=0x7f20440f7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:28:55.339 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.330+0000 7f2048a16700 1 -- 172.21.15.58:0/4049157387 shutdown_connections 2021-11-20T05:28:55.339 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.330+0000 7f2048a16700 1 -- 172.21.15.58:0/4049157387 wait complete. 2021-11-20T05:28:55.339 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.331+0000 7f2048a16700 1 Processor -- start 2021-11-20T05:28:55.340 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.331+0000 7f2048a16700 1 -- start start 2021-11-20T05:28:55.340 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.331+0000 7f2048a16700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f20440fc000 0x7f204410c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:28:55.340 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.332+0000 7f2048a16700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204410c830 con 0x7f20440fc000 2021-11-20T05:28:55.340 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.332+0000 7f20437fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f20440fc000 0x7f204410c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:28:55.341 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.332+0000 7f20437fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f20440fc000 0x7f204410c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39496/0 (socket says 172.21.15.58:39496) 2021-11-20T05:28:55.341 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.332+0000 7f20437fe700 1 -- 172.21.15.58:0/4150760969 learned_addr learned my addr 172.21.15.58:0/4150760969 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:28:55.341 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.332+0000 7f20437fe700 1 -- 172.21.15.58:0/4150760969 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203400b200 con 0x7f20440fc000 2021-11-20T05:28:55.341 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.332+0000 7f20437fe700 1 --2- 172.21.15.58:0/4150760969 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f20440fc000 0x7f204410c340 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f2034006c70 tx=0x7f2034016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:28:55.342 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.332+0000 7f202bfff700 1 -- 172.21.15.58:0/4150760969 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2034011b40 con 0x7f20440fc000 2021-11-20T05:28:55.342 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.332+0000 7f202bfff700 1 -- 172.21.15.58:0/4150760969 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2034011ca0 con 0x7f20440fc000 2021-11-20T05:28:55.342 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.332+0000 7f202bfff700 1 -- 172.21.15.58:0/4150760969 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f203401d430 con 0x7f20440fc000 2021-11-20T05:28:55.342 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.332+0000 7f2048a16700 1 -- 172.21.15.58:0/4150760969 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204410ca30 con 0x7f20440fc000 2021-11-20T05:28:55.343 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.332+0000 7f2048a16700 1 -- 172.21.15.58:0/4150760969 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f204410ce40 con 0x7f20440fc000 2021-11-20T05:28:55.343 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.333+0000 7f202bfff700 1 -- 172.21.15.58:0/4150760969 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f203400f070 con 0x7f20440fc000 2021-11-20T05:28:55.343 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.333+0000 7f202bfff700 1 -- 172.21.15.58:0/4150760969 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f20340277d0 con 0x7f20440fc000 2021-11-20T05:28:55.344 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.334+0000 7f2029ffb700 1 -- 172.21.15.58:0/4150760969 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f201c004fa0 con 0x7f20440fc000 2021-11-20T05:28:55.344 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.336+0000 7f202bfff700 1 -- 172.21.15.58:0/4150760969 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2034016070 con 0x7f20440fc000 2021-11-20T05:28:55.461 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.460+0000 7f2029ffb700 1 -- 172.21.15.58:0/4150760969 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f201c005940 con 0x7f20440fc000 2021-11-20T05:28:55.462 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.461+0000 7f202bfff700 1 -- 172.21.15.58:0/4150760969 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f203401d770 con 0x7f20440fc000 2021-11-20T05:28:55.462 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: cluster: 2021-11-20T05:28:55.462 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: id: b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:28:55.463 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: health: HEALTH_OK 2021-11-20T05:28:55.463 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.464 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: services: 2021-11-20T05:28:55.464 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi058 (age 0.674833s) 2021-11-20T05:28:55.464 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: mgr: no daemons active 2021-11-20T05:28:55.464 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2021-11-20T05:28:55.465 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.465 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: data: 2021-11-20T05:28:55.465 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2021-11-20T05:28:55.466 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2021-11-20T05:28:55.466 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2021-11-20T05:28:55.466 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: pgs: 2021-11-20T05:28:55.466 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.467 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.461+0000 7f2029ffb700 1 -- 172.21.15.58:0/4150760969 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f20440fc000 msgr2=0x7f204410c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:28:55.467 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.461+0000 7f2029ffb700 1 --2- 172.21.15.58:0/4150760969 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f20440fc000 0x7f204410c340 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f2034006c70 tx=0x7f2034016000).stop 2021-11-20T05:28:55.467 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.462+0000 7f2029ffb700 1 -- 172.21.15.58:0/4150760969 shutdown_connections 2021-11-20T05:28:55.467 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.462+0000 7f2029ffb700 1 --2- 172.21.15.58:0/4150760969 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f20440fc000 0x7f204410c340 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:28:55.468 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.462+0000 7f2029ffb700 1 -- 172.21.15.58:0/4150760969 >> 172.21.15.58:0/4150760969 conn(0x7f20440f5870 msgr2=0x7f20440f7590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:28:55.468 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.462+0000 7f2029ffb700 1 -- 172.21.15.58:0/4150760969 shutdown_connections 2021-11-20T05:28:55.468 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:55.462+0000 7f2029ffb700 1 -- 172.21.15.58:0/4150760969 wait complete. 2021-11-20T05:28:56.098 INFO:teuthology.orchestra.run.smithi058.stderr:mon is available 2021-11-20T05:28:56.098 INFO:teuthology.orchestra.run.smithi058.stderr:Assimilating anything we can from ceph.conf... 2021-11-20T05:28:56.099 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z -v /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058:/var/lib/ceph/mon/ceph-smithi058:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi058/config 2021-11-20T05:28:56.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:55 smithi058 conmon[32845]: cluster 2021-11-20T05:28: 2021-11-20T05:28:56.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:55 smithi058 conmon[32845]: 54.787033+0000 mon.smithi058 (mon.0) 2 : cluster [INF] mon.smithi058 is new leader, mons smithi058 in quorum (ranks 0) 2021-11-20T05:28:56.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:55 smithi058 conmon[32845]: cluster 2021-11-20T05:28:54.787153+0000 mon.smithi058 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi058=[v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0]} 2021-11-20T05:28:56.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:55 smithi058 conmon[32845]: cluster 2021-11-20T05:28:54.788541+0000 mon.smithi058 (mon.0) 4 : cluster [DBG] fsmap 2021-11-20T05:28:56.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:55 smithi058 conmon[32845]: cluster 2021-11-20T05:28:54.790345+0000 mon.smithi058 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-11-20T05:28:56.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:55 smithi058 conmon[32845]: cluster 2021-11-20T05:28:54.790684+0000 mon.smithi058 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2021-11-20T05:28:56.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:55 smithi058 conmon[32845]: audit 2021-11-20T05:28:55.461819+0000 mon.smithi058 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.58:0/4150760969' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2021-11-20T05:28:56.625 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.621+0000 7f89aa3f4700 1 Processor -- start 2021-11-20T05:28:56.625 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.621+0000 7f89aa3f4700 1 -- start start 2021-11-20T05:28:56.626 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.621+0000 7f89aa3f4700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89a40fbbc0 0x7f89a40fbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:28:56.626 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.621+0000 7f89aa3f4700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40fc4d0 con 0x7f89a40fbbc0 2021-11-20T05:28:56.626 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.621+0000 7f89a3fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89a40fbbc0 0x7f89a40fbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:28:56.627 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.621+0000 7f89a3fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89a40fbbc0 0x7f89a40fbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39498/0 (socket says 172.21.15.58:39498) 2021-11-20T05:28:56.627 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.621+0000 7f89a3fff700 1 -- 172.21.15.58:0/1583182564 learned_addr learned my addr 172.21.15.58:0/1583182564 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:28:56.628 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.622+0000 7f89a3fff700 1 -- 172.21.15.58:0/1583182564 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a40fc610 con 0x7f89a40fbbc0 2021-11-20T05:28:56.628 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.622+0000 7f89a3fff700 1 --2- 172.21.15.58:0/1583182564 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89a40fbbc0 0x7f89a40fbfe0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f898c00b5f0 tx=0x7f898c006ee0).ready entity=mon.0 client_cookie=2305b4f98b42a3d8 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:28:56.629 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.622+0000 7f89a2ffd700 1 -- 172.21.15.58:0/1583182564 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f898c00d600 con 0x7f89a40fbbc0 2021-11-20T05:28:56.629 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.622+0000 7f89a2ffd700 1 -- 172.21.15.58:0/1583182564 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f898c00d760 con 0x7f89a40fbbc0 2021-11-20T05:28:56.629 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.622+0000 7f89a2ffd700 1 -- 172.21.15.58:0/1583182564 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f898c00da30 con 0x7f89a40fbbc0 2021-11-20T05:28:56.630 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.622+0000 7f89aa3f4700 1 -- 172.21.15.58:0/1583182564 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89a40fbbc0 msgr2=0x7f89a40fbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:28:56.630 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.622+0000 7f89aa3f4700 1 --2- 172.21.15.58:0/1583182564 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89a40fbbc0 0x7f89a40fbfe0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f898c00b5f0 tx=0x7f898c006ee0).stop 2021-11-20T05:28:56.630 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.622+0000 7f89aa3f4700 1 -- 172.21.15.58:0/1583182564 shutdown_connections 2021-11-20T05:28:56.630 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.622+0000 7f89aa3f4700 1 --2- 172.21.15.58:0/1583182564 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89a40fbbc0 0x7f89a40fbfe0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:28:56.631 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.622+0000 7f89aa3f4700 1 -- 172.21.15.58:0/1583182564 >> 172.21.15.58:0/1583182564 conn(0x7f89a40f73e0 msgr2=0x7f89a40f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:28:56.631 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.623+0000 7f89aa3f4700 1 -- 172.21.15.58:0/1583182564 shutdown_connections 2021-11-20T05:28:56.631 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.623+0000 7f89aa3f4700 1 -- 172.21.15.58:0/1583182564 wait complete. 2021-11-20T05:28:56.631 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.623+0000 7f89aa3f4700 1 Processor -- start 2021-11-20T05:28:56.632 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.624+0000 7f89aa3f4700 1 -- start start 2021-11-20T05:28:56.632 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.624+0000 7f89aa3f4700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89a40fbbc0 0x7f89a410a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:28:56.632 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.624+0000 7f89aa3f4700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a410a6b0 con 0x7f89a40fbbc0 2021-11-20T05:28:56.632 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.624+0000 7f89a3fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89a40fbbc0 0x7f89a410a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:28:56.632 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.624+0000 7f89a3fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89a40fbbc0 0x7f89a410a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39500/0 (socket says 172.21.15.58:39500) 2021-11-20T05:28:56.633 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.624+0000 7f89a3fff700 1 -- 172.21.15.58:0/114074910 learned_addr learned my addr 172.21.15.58:0/114074910 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:28:56.633 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.624+0000 7f89a3fff700 1 -- 172.21.15.58:0/114074910 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898c00b200 con 0x7f89a40fbbc0 2021-11-20T05:28:56.633 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.624+0000 7f89a3fff700 1 --2- 172.21.15.58:0/114074910 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89a40fbbc0 0x7f89a410a1c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f898c006c70 tx=0x7f898c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:28:56.634 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.624+0000 7f89a0ff9700 1 -- 172.21.15.58:0/114074910 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f898c011b40 con 0x7f89a40fbbc0 2021-11-20T05:28:56.634 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.625+0000 7f89a0ff9700 1 -- 172.21.15.58:0/114074910 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f898c011ca0 con 0x7f89a40fbbc0 2021-11-20T05:28:56.634 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.625+0000 7f89aa3f4700 1 -- 172.21.15.58:0/114074910 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a410a8b0 con 0x7f89a40fbbc0 2021-11-20T05:28:56.634 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.625+0000 7f89a0ff9700 1 -- 172.21.15.58:0/114074910 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f898c01d430 con 0x7f89a40fbbc0 2021-11-20T05:28:56.635 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.625+0000 7f89aa3f4700 1 -- 172.21.15.58:0/114074910 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a410acc0 con 0x7f89a40fbbc0 2021-11-20T05:28:56.635 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.625+0000 7f89a0ff9700 1 -- 172.21.15.58:0/114074910 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f898c00f070 con 0x7f89a40fbbc0 2021-11-20T05:28:56.635 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.625+0000 7f89a0ff9700 1 -- 172.21.15.58:0/114074910 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f898c027460 con 0x7f89a40fbbc0 2021-11-20T05:28:56.636 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.626+0000 7f89aa3f4700 1 -- 172.21.15.58:0/114074910 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a4045ad0 con 0x7f89a40fbbc0 2021-11-20T05:28:56.636 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.628+0000 7f89a0ff9700 1 -- 172.21.15.58:0/114074910 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f898c016070 con 0x7f89a40fbbc0 2021-11-20T05:28:56.748 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.749+0000 7f89aa3f4700 1 -- 172.21.15.58:0/114074910 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f89a40fcf00 con 0x7f89a40fbbc0 2021-11-20T05:28:56.751 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.752+0000 7f89a0ff9700 1 -- 172.21.15.58:0/114074910 <== mon.0 v2:172.21.15.58:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f898c01d620 con 0x7f89a40fbbc0 2021-11-20T05:28:56.752 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.752+0000 7f89a0ff9700 1 -- 172.21.15.58:0/114074910 <== mon.0 v2:172.21.15.58:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7f898c030840 con 0x7f89a40fbbc0 2021-11-20T05:28:56.755 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.755 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: [global] 2021-11-20T05:28:56.755 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: fsid = b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:28:56.755 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.58:3300,v1:172.21.15.58:6789] 2021-11-20T05:28:56.756 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2021-11-20T05:28:56.756 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2021-11-20T05:28:56.756 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2021-11-20T05:28:56.756 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2021-11-20T05:28:56.756 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.757 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: [mgr] 2021-11-20T05:28:56.757 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2021-11-20T05:28:56.757 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.757 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: [osd] 2021-11-20T05:28:56.758 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2021-11-20T05:28:56.758 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2021-11-20T05:28:56.758 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2021-11-20T05:28:56.759 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2021-11-20T05:28:56.759 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2021-11-20T05:28:56.759 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2021-11-20T05:28:56.759 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.753+0000 7f89aa3f4700 1 -- 172.21.15.58:0/114074910 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89a40fbbc0 msgr2=0x7f89a410a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:28:56.759 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.753+0000 7f89aa3f4700 1 --2- 172.21.15.58:0/114074910 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89a40fbbc0 0x7f89a410a1c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f898c006c70 tx=0x7f898c016000).stop 2021-11-20T05:28:56.760 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.753+0000 7f89aa3f4700 1 -- 172.21.15.58:0/114074910 shutdown_connections 2021-11-20T05:28:56.760 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.753+0000 7f89aa3f4700 1 --2- 172.21.15.58:0/114074910 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89a40fbbc0 0x7f89a410a1c0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:28:56.760 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.753+0000 7f89aa3f4700 1 -- 172.21.15.58:0/114074910 >> 172.21.15.58:0/114074910 conn(0x7f89a40f73e0 msgr2=0x7f89a40fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:28:56.760 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.753+0000 7f89aa3f4700 1 -- 172.21.15.58:0/114074910 shutdown_connections 2021-11-20T05:28:56.761 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:56.753+0000 7f89aa3f4700 1 -- 172.21.15.58:0/114074910 wait complete. 2021-11-20T05:28:56.905 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:56 smithi058 conmon[32845]: audit 2021-11-20T 2021-11-20T05:28:56.905 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:56 smithi058 conmon[32845]: 05:28:56.750127+0000 mon.smithi058 (mon.0) 8 : audit [INF] from='client.? 172.21.15.58:0/114074910' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-11-20T05:28:56.906 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:56 smithi058 conmon[32845]: audit 2021-11-20T05:28:56.753192+0000 mon.smithi058 (mon.0) 9 : audit [INF] 2021-11-20T05:28:56.906 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:56 smithi058 conmon[32845]: from='client.? 172.21.15.58:0/114074910' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2021-11-20T05:28:57.137 INFO:teuthology.orchestra.run.smithi058.stderr:Generating new minimal ceph.conf... 2021-11-20T05:28:57.137 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z -v /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058:/var/lib/ceph/mon/ceph-smithi058:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi058/config 2021-11-20T05:28:57.607 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.604+0000 7f221eec3700 1 Processor -- start 2021-11-20T05:28:57.607 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.604+0000 7f221eec3700 1 -- start start 2021-11-20T05:28:57.607 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.604+0000 7f221eec3700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f221806fbb0 0x7f2218100a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:28:57.607 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.604+0000 7f221eec3700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22180fc470 con 0x7f221806fbb0 2021-11-20T05:28:57.608 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.605+0000 7f221dec1700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f221806fbb0 0x7f2218100a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:28:57.608 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.605+0000 7f221dec1700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f221806fbb0 0x7f2218100a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39502/0 (socket says 172.21.15.58:39502) 2021-11-20T05:28:57.608 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.605+0000 7f221dec1700 1 -- 172.21.15.58:0/1237224463 learned_addr learned my addr 172.21.15.58:0/1237224463 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:28:57.608 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.605+0000 7f221dec1700 1 -- 172.21.15.58:0/1237224463 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2218100f70 con 0x7f221806fbb0 2021-11-20T05:28:57.609 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.605+0000 7f221dec1700 1 --2- 172.21.15.58:0/1237224463 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f221806fbb0 0x7f2218100a80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f221400b5f0 tx=0x7f2214006ee0).ready entity=mon.0 client_cookie=17019190f3f1a02a server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:28:57.609 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.605+0000 7f221cebf700 1 -- 172.21.15.58:0/1237224463 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f221400d600 con 0x7f221806fbb0 2021-11-20T05:28:57.609 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.605+0000 7f221cebf700 1 -- 172.21.15.58:0/1237224463 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f221400dbd0 con 0x7f221806fbb0 2021-11-20T05:28:57.609 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.605+0000 7f221cebf700 1 -- 172.21.15.58:0/1237224463 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2214011c00 con 0x7f221806fbb0 2021-11-20T05:28:57.610 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.606+0000 7f221eec3700 1 -- 172.21.15.58:0/1237224463 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f221806fbb0 msgr2=0x7f2218100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:28:57.611 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.606+0000 7f221eec3700 1 --2- 172.21.15.58:0/1237224463 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f221806fbb0 0x7f2218100a80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f221400b5f0 tx=0x7f2214006ee0).stop 2021-11-20T05:28:57.611 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.606+0000 7f221eec3700 1 -- 172.21.15.58:0/1237224463 shutdown_connections 2021-11-20T05:28:57.611 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.606+0000 7f221eec3700 1 --2- 172.21.15.58:0/1237224463 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f221806fbb0 0x7f2218100a80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:28:57.611 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.606+0000 7f221eec3700 1 -- 172.21.15.58:0/1237224463 >> 172.21.15.58:0/1237224463 conn(0x7f22180f60e0 msgr2=0x7f22180f8500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:28:57.612 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.606+0000 7f221eec3700 1 -- 172.21.15.58:0/1237224463 shutdown_connections 2021-11-20T05:28:57.612 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.606+0000 7f221eec3700 1 -- 172.21.15.58:0/1237224463 wait complete. 2021-11-20T05:28:57.612 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.607+0000 7f221eec3700 1 Processor -- start 2021-11-20T05:28:57.612 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.607+0000 7f221eec3700 1 -- start start 2021-11-20T05:28:57.612 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.607+0000 7f221eec3700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f221806fbb0 0x7f221810c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:28:57.613 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.607+0000 7f221eec3700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221810ce80 con 0x7f221806fbb0 2021-11-20T05:28:57.613 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.607+0000 7f221dec1700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f221806fbb0 0x7f221810c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:28:57.613 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.607+0000 7f221dec1700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f221806fbb0 0x7f221810c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39504/0 (socket says 172.21.15.58:39504) 2021-11-20T05:28:57.613 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.607+0000 7f221dec1700 1 -- 172.21.15.58:0/937674573 learned_addr learned my addr 172.21.15.58:0/937674573 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:28:57.614 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.607+0000 7f221dec1700 1 -- 172.21.15.58:0/937674573 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221400b200 con 0x7f221806fbb0 2021-11-20T05:28:57.615 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.607+0000 7f221dec1700 1 --2- 172.21.15.58:0/937674573 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f221806fbb0 0x7f221810c990 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f221400b190 tx=0x7f2214006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:28:57.615 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.608+0000 7f220a7fc700 1 -- 172.21.15.58:0/937674573 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f221401b410 con 0x7f221806fbb0 2021-11-20T05:28:57.615 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.608+0000 7f220a7fc700 1 -- 172.21.15.58:0/937674573 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f221401b9e0 con 0x7f221806fbb0 2021-11-20T05:28:57.616 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.608+0000 7f221eec3700 1 -- 172.21.15.58:0/937674573 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f221810d080 con 0x7f221806fbb0 2021-11-20T05:28:57.616 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.608+0000 7f220a7fc700 1 -- 172.21.15.58:0/937674573 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f221401a880 con 0x7f221806fbb0 2021-11-20T05:28:57.616 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.608+0000 7f221eec3700 1 -- 172.21.15.58:0/937674573 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f221810d490 con 0x7f221806fbb0 2021-11-20T05:28:57.616 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.608+0000 7f220a7fc700 1 -- 172.21.15.58:0/937674573 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f221400f070 con 0x7f221806fbb0 2021-11-20T05:28:57.617 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.608+0000 7f220a7fc700 1 -- 172.21.15.58:0/937674573 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2214025c90 con 0x7f221806fbb0 2021-11-20T05:28:57.617 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.609+0000 7f221eec3700 1 -- 172.21.15.58:0/937674573 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2218045ad0 con 0x7f221806fbb0 2021-11-20T05:28:57.617 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.611+0000 7f220a7fc700 1 -- 172.21.15.58:0/937674573 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2214016070 con 0x7f221806fbb0 2021-11-20T05:28:57.732 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.732+0000 7f221eec3700 1 -- 172.21.15.58:0/937674573 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f22181067a0 con 0x7f221806fbb0 2021-11-20T05:28:57.732 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.733+0000 7f220a7fc700 1 -- 172.21.15.58:0/937674573 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f221401bb50 con 0x7f221806fbb0 2021-11-20T05:28:57.734 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.733+0000 7f221eec3700 1 -- 172.21.15.58:0/937674573 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f221806fbb0 msgr2=0x7f221810c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:28:57.734 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.733+0000 7f221eec3700 1 --2- 172.21.15.58:0/937674573 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f221806fbb0 0x7f221810c990 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f221400b190 tx=0x7f2214006ee0).stop 2021-11-20T05:28:57.734 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.734+0000 7f221eec3700 1 -- 172.21.15.58:0/937674573 shutdown_connections 2021-11-20T05:28:57.734 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.734+0000 7f221eec3700 1 --2- 172.21.15.58:0/937674573 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f221806fbb0 0x7f221810c990 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:28:57.735 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.734+0000 7f221eec3700 1 -- 172.21.15.58:0/937674573 >> 172.21.15.58:0/937674573 conn(0x7f22180f60e0 msgr2=0x7f22180f7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:28:57.735 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.734+0000 7f221eec3700 1 -- 172.21.15.58:0/937674573 shutdown_connections 2021-11-20T05:28:57.735 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:57.734+0000 7f221eec3700 1 -- 172.21.15.58:0/937674573 wait complete. 2021-11-20T05:28:57.897 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:57 smithi058 conmon[32845]: audit 2021-11-20T05:28:57.733793+0000 mon.smithi058 (mon.0 2021-11-20T05:28:57.897 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:57 smithi058 conmon[32845]: ) 10 : audit [DBG] from='client.? 172.21.15.58:0/937674573' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:28:58.074 INFO:teuthology.orchestra.run.smithi058.stderr:Restarting the monitor... 2021-11-20T05:28:58.074 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl restart ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mon.smithi058 2021-11-20T05:28:58.153 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:58 smithi058 systemd[1]: Stopping Ceph mon.smithi058 for b33f8c4a-49c2-11ec-8c2c-001a4aab830c... 2021-11-20T05:28:58.154 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:58 smithi058 conmon[32845]: debug 2021-11-20T05:28:58.153+0000 7f61ab054700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi058 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-11-20T05:28:58.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:58 smithi058 conmon[32845]: debug 2021-11-20T05:28:58.153+0000 7f61ab054700 -1 mon.smithi058@0(leader) e1 *** Got Signal Terminated *** 2021-11-20T05:28:58.953 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:58 smithi058 podman[33327]: ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c-mon.smithi058 2021-11-20T05:28:58.953 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:58 smithi058 systemd[1]: ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mon.smithi058.service: Succeeded. 2021-11-20T05:28:58.954 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:58 smithi058 systemd[1]: Stopped Ceph mon.smithi058 for b33f8c4a-49c2-11ec-8c2c-001a4aab830c. 2021-11-20T05:28:58.954 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:58 smithi058 systemd[1]: Starting Ceph mon.smithi058 for b33f8c4a-49c2-11ec-8c2c-001a4aab830c... 2021-11-20T05:28:59.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:58 smithi058 podman[33497]: 2021-11-20T05:28:59.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.142+0000 7ff1bc8de700 0 set uid:gid to 167:167 (ceph:ceph) 2021-11-20T05:28:59.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.142+0000 7ff1bc8de700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2021-11-20T05:28:59.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.142+0000 7ff1bc8de700 0 pidfile_write: ignore empty --pid-file 2021-11-20T05:28:59.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.146+0000 7ff1bc8de700 0 load: jerasure load: lrc load: isa 2021-11-20T05:28:59.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: RocksDB version: 6.8.1 2021-11-20T05:28:59.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-11-20T05:28:59.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.223 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Compile date May 13 2021 2021-11-20T05:28:59.223 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: DB SUMMARY 2021-11-20T05:28:59.223 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.223 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: CURRENT file: CURRENT 2021-11-20T05:28:59.224 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.224 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: IDENTITY file: IDENTITY 2021-11-20T05:28:59.224 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.225 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2021-11-20T05:28:59.225 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.225 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi058/store.db dir, Total Num: 1, files: 000004.sst 2021-11-20T05:28:59.225 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.225 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi058/store.db: 000006.log size: 97725 ; 2021-11-20T05:28:59.226 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.226 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.error_if_exists: 0 2021-11-20T05:28:59.226 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.create_if_missing: 0 2021-11-20T05:28:59.226 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.paranoid_checks: 1 2021-11-20T05:28:59.227 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.env: 0x55d5cd2e31c0 2021-11-20T05:28:59.227 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.227 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.fs: Posix File System 2021-11-20T05:28:59.228 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.info_log: 0x55d5ce5a3960 2021-11-20T05:28:59.228 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_file_opening_threads: 16 2021-11-20T05:28:59.228 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.statistics: (nil) 2021-11-20T05:28:59.229 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.use_fsync: 0 2021-11-20T05:28:59.229 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_log_file_size: 0 2021-11-20T05:28:59.229 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-11-20T05:28:59.229 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-11-20T05:28:59.229 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.keep_log_file_num: 1000 2021-11-20T05:28:59.230 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.recycle_log_file_num: 0 2021-11-20T05:28:59.230 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.allow_fallocate: 1 2021-11-20T05:28:59.230 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.allow_mmap_reads: 0 2021-11-20T05:28:59.231 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.allow_mmap_writes: 0 2021-11-20T05:28:59.231 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.use_direct_reads: 0 2021-11-20T05:28:59.231 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-11-20T05:28:59.231 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.create_missing_column_families: 0 2021-11-20T05:28:59.232 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.db_log_dir: 2021-11-20T05:28:59.232 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi058/store.db 2021-11-20T05:28:59.232 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-11-20T05:28:59.232 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.232 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_subcompactions: 1 2021-11-20T05:28:59.233 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_background_flushes: -1 2021-11-20T05:28:59.233 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-11-20T05:28:59.233 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-11-20T05:28:59.233 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-11-20T05:28:59.234 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-11-20T05:28:59.234 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-11-20T05:28:59.234 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.advise_random_on_open: 1 2021-11-20T05:28:59.234 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.db_write_buffer_size: 0 2021-11-20T05:28:59.235 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.write_buffer_manager: 0x55d5cf218030 2021-11-20T05:28:59.235 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-11-20T05:28:59.235 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-11-20T05:28:59.236 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-11-20T05:28:59.236 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-11-20T05:28:59.236 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.rate_limiter: (nil) 2021-11-20T05:28:59.236 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-11-20T05:28:59.237 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.wal_recovery_mode: 2 2021-11-20T05:28:59.237 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.enable_thread_tracking: 0 2021-11-20T05:28:59.237 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.enable_pipelined_write: 0 2021-11-20T05:28:59.237 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.unordered_write: 0 2021-11-20T05:28:59.237 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-11-20T05:28:59.238 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-11-20T05:28:59.238 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-11-20T05:28:59.238 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-11-20T05:28:59.238 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.row_cache: None 2021-11-20T05:28:59.239 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.wal_filter: None 2021-11-20T05:28:59.239 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-11-20T05:28:59.239 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.allow_ingest_behind: 0 2021-11-20T05:28:59.239 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.preserve_deletes: 0 2021-11-20T05:28:59.240 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.240 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.two_write_queues: 0 2021-11-20T05:28:59.240 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.manual_wal_flush: 0 2021-11-20T05:28:59.241 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.atomic_flush: 0 2021-11-20T05:28:59.241 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-11-20T05:28:59.241 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-11-20T05:28:59.242 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-11-20T05:28:59.242 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.log_readahead_size: 0 2021-11-20T05:28:59.242 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-11-20T05:28:59.242 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_background_jobs: 2 2021-11-20T05:28:59.242 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_background_compactions: -1 2021-11-20T05:28:59.243 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-11-20T05:28:59.243 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-11-20T05:28:59.243 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-11-20T05:28:59.244 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_total_wal_size: 0 2021-11-20T05:28:59.244 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-11-20T05:28:59.244 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-11-20T05:28:59.244 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-11-20T05:28:59.244 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-11-20T05:28:59.245 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_open_files: -1 2021-11-20T05:28:59.245 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.bytes_per_sync: 0 2021-11-20T05:28:59.245 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-11-20T05:28:59.246 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-11-20T05:28:59.246 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compaction_readahead_size: 0 2021-11-20T05:28:59.247 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Compression algorithms supported: 2021-11-20T05:28:59.247 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-11-20T05:28:59.247 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: kZSTD supported: 0 2021-11-20T05:28:59.248 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: kXpressCompression supported: 0 2021-11-20T05:28:59.248 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: kLZ4HCCompression supported: 1 2021-11-20T05:28:59.248 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: kLZ4Compression supported: 1 2021-11-20T05:28:59.249 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: kBZip2Compression supported: 0 2021-11-20T05:28:59.249 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: kZlibCompression supported: 1 2021-11-20T05:28:59.249 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: kSnappyCompression supported: 1 2021-11-20T05:28:59.249 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-11-20T05:28:59.250 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi058/store.db/MANIFEST-000005 2021-11-20T05:28:59.250 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.250 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-11-20T05:28:59.250 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.251 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-11-20T05:28:59.251 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.merge_operator: 2021-11-20T05:28:59.251 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compaction_filter: None 2021-11-20T05:28:59.252 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compaction_filter_factory: None 2021-11-20T05:28:59.252 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-11-20T05:28:59.252 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-11-20T05:28:59.252 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d5ce4ba128) 2021-11-20T05:28:59.253 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: cache_index_and_filter_blocks: 1 2021-11-20T05:28:59.253 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: cache_index_and_filter_blocks_with_high_priority: 0 2021-11-20T05:28:59.253 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-11-20T05:28:59.253 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: pin_top_level_index_and_filter: 1 2021-11-20T05:28:59.253 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: index_type: 0 2021-11-20T05:28:59.254 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: data_block_index_type: 0 2021-11-20T05:28:59.254 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: index_shortening: 1 2021-11-20T05:28:59.254 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: data_block_hash_table_util_ratio: 0.750000 2021-11-20T05:28:59.254 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: hash_index_allow_collision: 1 2021-11-20T05:28:59.255 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: checksum: 1 2021-11-20T05:28:59.255 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: no_block_cache: 0 2021-11-20T05:28:59.255 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: block_cache: 0x55d5ce4f0f10 2021-11-20T05:28:59.255 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: block_cache_name: BinnedLRUCache 2021-11-20T05:28:59.256 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: block_cache_options: 2021-11-20T05:28:59.256 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: capacity : 536870912 2021-11-20T05:28:59.256 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: num_shard_bits : 4 2021-11-20T05:28:59.256 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: strict_capacity_limit : 0 2021-11-20T05:28:59.257 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: high_pri_pool_ratio: 0.000 2021-11-20T05:28:59.257 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: block_cache_compressed: (nil) 2021-11-20T05:28:59.257 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: persistent_cache: (nil) 2021-11-20T05:28:59.257 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: block_size: 4096 2021-11-20T05:28:59.257 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: block_size_deviation: 10 2021-11-20T05:28:59.258 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: block_restart_interval: 16 2021-11-20T05:28:59.258 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: index_block_restart_interval: 1 2021-11-20T05:28:59.258 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: metadata_block_size: 4096 2021-11-20T05:28:59.258 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: partition_filters: 0 2021-11-20T05:28:59.259 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: use_delta_encoding: 1 2021-11-20T05:28:59.259 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: filter_policy: rocksdb.BuiltinBloomFilter 2021-11-20T05:28:59.259 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: whole_key_filtering: 1 2021-11-20T05:28:59.259 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: verify_compression: 0 2021-11-20T05:28:59.259 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: read_amp_bytes_per_bit: 0 2021-11-20T05:28:59.260 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: format_version: 2 2021-11-20T05:28:59.260 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: enable_index_compression: 1 2021-11-20T05:28:59.260 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: block_align: 0 2021-11-20T05:28:59.260 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.261 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.write_buffer_size: 33554432 2021-11-20T05:28:59.261 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_write_buffer_number: 2 2021-11-20T05:28:59.261 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compression: NoCompression 2021-11-20T05:28:59.261 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.bottommost_compression: Disabled 2021-11-20T05:28:59.261 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.prefix_extractor: nullptr 2021-11-20T05:28:59.262 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-11-20T05:28:59.262 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.num_levels: 7 2021-11-20T05:28:59.262 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-11-20T05:28:59.262 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-11-20T05:28:59.263 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-11-20T05:28:59.263 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-11-20T05:28:59.263 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-11-20T05:28:59.263 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-11-20T05:28:59.264 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-11-20T05:28:59.264 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-11-20T05:28:59.264 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-11-20T05:28:59.264 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-11-20T05:28:59.265 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compression_opts.level: 32767 2021-11-20T05:28:59.265 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compression_opts.strategy: 0 2021-11-20T05:28:59.265 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-11-20T05:28:59.265 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-11-20T05:28:59.265 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compression_opts.enabled: false 2021-11-20T05:28:59.266 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-11-20T05:28:59.266 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-11-20T05:28:59.266 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-11-20T05:28:59.266 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.target_file_size_base: 67108864 2021-11-20T05:28:59.267 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-11-20T05:28:59.267 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-11-20T05:28:59.267 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-11-20T05:28:59.267 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-11-20T05:28:59.267 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-11-20T05:28:59.268 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-11-20T05:28:59.268 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-11-20T05:28:59.268 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-11-20T05:28:59.268 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-11-20T05:28:59.269 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-11-20T05:28:59.269 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-11-20T05:28:59.269 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-11-20T05:28:59.269 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-11-20T05:28:59.270 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.arena_block_size: 4194304 2021-11-20T05:28:59.270 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-11-20T05:28:59.270 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-11-20T05:28:59.270 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-11-20T05:28:59.270 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.disable_auto_compactions: 0 2021-11-20T05:28:59.271 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-11-20T05:28:59.271 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-11-20T05:28:59.271 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-11-20T05:28:59.271 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-11-20T05:28:59.272 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-11-20T05:28:59.272 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-11-20T05:28:59.272 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-11-20T05:28:59.272 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-11-20T05:28:59.273 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-11-20T05:28:59.273 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-11-20T05:28:59.273 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.table_properties_collectors: 2021-11-20T05:28:59.273 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.inplace_update_support: 0 2021-11-20T05:28:59.273 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-11-20T05:28:59.274 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-11-20T05:28:59.274 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-11-20T05:28:59.274 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-11-20T05:28:59.276 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.bloom_locality: 0 2021-11-20T05:28:59.276 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.max_successive_merges: 0 2021-11-20T05:28:59.276 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-11-20T05:28:59.276 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.paranoid_file_checks: 0 2021-11-20T05:28:59.277 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.force_consistency_checks: 0 2021-11-20T05:28:59.277 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.report_bg_io_stats: 0 2021-11-20T05:28:59.277 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.ttl: 2592000 2021-11-20T05:28:59.277 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.147+0000 7ff1bc8de700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-11-20T05:28:59.278 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.148+0000 7ff1bc8de700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi058/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-11-20T05:28:59.278 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.278 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.148+0000 7ff1bc8de700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2021-11-20T05:28:59.278 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.279 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.148+0000 7ff1bc8de700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637386139149146, "job": 1, "event": "recovery_started", "log_files": [6]} 2021-11-20T05:28:59.279 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.148+0000 7ff1bc8de700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2021-11-20T05:28:59.279 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.148+0000 7ff1bc8de700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-11-20T05:28:59.279 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.149+0000 7ff1bc8de700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637386139150033, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 93395, "table_properties": {"data_size": 91495, "index_size": 377, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12317, "raw_average_key_size": 49, "raw_value_size": 84651, "raw_average_value_size": 338, "num_data_blocks": 12, "num_entries": 250, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637386139, "oldest_key_time": 3, "file_creation_time": 0}} 2021-11-20T05:28:59.279 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.149+0000 7ff1bc8de700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2021-11-20T05:28:59.280 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.280 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.149+0000 7ff1bc8de700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637386139150414, "job": 1, "event": "recovery_finished"} 2021-11-20T05:28:59.280 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.150+0000 7ff1bc8de700 4 rocksdb: DB pointer 0x55d5ce5b5800 2021-11-20T05:28:59.280 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.150+0000 7ff1a36f6700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-11-20T05:28:59.281 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.150+0000 7ff1a36f6700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-11-20T05:28:59.281 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: ** DB Stats ** 2021-11-20T05:28:59.281 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T05:28:59.281 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-11-20T05:28:59.282 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-11-20T05:28:59.282 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T05:28:59.282 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-11-20T05:28:59.282 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-11-20T05:28:59.283 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T05:28:59.283 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.283 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: ** Compaction Stats [default] ** 2021-11-20T05:28:59.283 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:28:59.284 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:28:59.284 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: L0 2/0 92.90 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.2 0.00 0.00 1 0.001 0 0 2021-11-20T05:28:59.284 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Sum 2/0 92.90 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.2 0.00 0.00 1 0.001 0 0 2021-11-20T05:28:59.284 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.2 0.00 0.00 1 0.001 0 0 2021-11-20T05:28:59.284 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.285 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: ** Compaction Stats [default] ** 2021-11-20T05:28:59.285 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:28:59.285 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:28:59.285 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.2 0.00 0.00 1 0.001 0 0 2021-11-20T05:28:59.286 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T05:28:59.286 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T05:28:59.286 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T05:28:59.286 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T05:28:59.287 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T05:28:59.287 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: AddFile(Keys): cumulative 0, interval 0 2021-11-20T05:28:59.287 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Cumulative compaction: 0.00 GB write, 33.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:28:59.287 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Interval compaction: 0.00 GB write, 33.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:28:59.288 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T05:28:59.288 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.288 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: ** File Read Latency Histogram By Level [default] ** 2021-11-20T05:28:59.288 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.289 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: ** Compaction Stats [default] ** 2021-11-20T05:28:59.289 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:28:59.289 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:28:59.289 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: L0 2/0 92.90 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.2 0.00 0.00 1 0.001 0 0 2021-11-20T05:28:59.290 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Sum 2/0 92.90 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.2 0.00 0.00 1 0.001 0 0 2021-11-20T05:28:59.290 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T05:28:59.290 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.290 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: ** Compaction Stats [default] ** 2021-11-20T05:28:59.291 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:28:59.291 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:28:59.291 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.2 0.00 0.00 1 0.001 0 0 2021-11-20T05:28:59.291 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T05:28:59.291 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T05:28:59.292 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T05:28:59.292 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T05:28:59.292 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T05:28:59.292 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: AddFile(Keys): cumulative 0, interval 0 2021-11-20T05:28:59.293 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Cumulative compaction: 0.00 GB write, 32.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:28:59.293 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:28:59.293 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T05:28:59.293 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.294 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: ** File Read Latency Histogram By Level [default] ** 2021-11-20T05:28:59.294 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.294 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.150+0000 7ff1bc8de700 0 starting mon.smithi058 rank 0 at public addrs [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] at bind addrs [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi058 fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:28:59.294 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.151+0000 7ff1bc8de700 1 mon.smithi058@-1(???) e1 preinit fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:28:59.294 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.151+0000 7ff1bc8de700 0 mon.smithi058@-1(???).mds e1 new map 2021-11-20T05:28:59.295 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.151+0000 7ff1bc8de700 0 mon.smithi058@-1(???).mds e1 print_map 2021-11-20T05:28:59.295 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: e1 2021-11-20T05:28:59.295 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: enable_multiple, ever_enabled_multiple: 1,1 2021-11-20T05:28:59.295 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T05:28:59.296 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: legacy client fscid: -1 2021-11-20T05:28:59.296 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.296 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: No filesystems configured 2021-11-20T05:28:59.296 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: 2021-11-20T05:28:59.297 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.152+0000 7ff1bc8de700 0 mon.smithi058@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2021-11-20T05:28:59.297 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.152+0000 7ff1bc8de700 0 mon.smithi058@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T05:28:59.297 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.152+0000 7ff1bc8de700 0 mon.smithi058@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T05:28:59.297 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.152+0000 7ff1bc8de700 0 mon.smithi058@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T05:28:59.298 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: audit 2021-11-20T05:28 2021-11-20T05:28:59.298 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: :57.733793+0000 mon.smithi058 (mon.0) 10 : audit [DBG] 2021-11-20T05:28:59.298 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: from='client.? 172.21.15.58:0/937674573' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:28:59.298 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.152+0000 7ff1bc8de700 1 mon.smithi058@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2021-11-20T05:28:59.299 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.153+0000 7ff1bc8de700 20 expand_channel_meta expand map: {default=false} 2021-11-20T05:28:59.299 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.153+0000 7ff1bc8de700 20 expand_channel_meta from 'false' to 'false' 2021-11-20T05:28:59.299 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.153+0000 7ff1bc8de700 20 expand_channel_meta expanded map: {default=false} 2021-11-20T05:28:59.299 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.153+0000 7ff1bc8de700 20 expand_channel_meta expand map: {default=info} 2021-11-20T05:28:59.300 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.153+0000 7ff1bc8de700 20 expand_channel_meta from 'info' to 'info' 2021-11-20T05:28:59.300 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.153+0000 7ff1bc8de700 20 expand_channel_meta expanded map: {default=info} 2021-11-20T05:28:59.300 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.153+0000 7ff1bc8de700 20 expand_channel_meta expand map: {default=daemon} 2021-11-20T05:28:59.300 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.153+0000 7ff1bc8de700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-11-20T05:28:59.301 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.153+0000 7ff1bc8de700 20 expand_channel_meta expanded map: {default=daemon} 2021-11-20T05:28:59.301 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.153+0000 7ff1bc8de700 20 expand_channel_meta expand map: {default=debug} 2021-11-20T05:28:59.301 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.153+0000 7ff1bc8de700 20 expand_channel_meta from 'debug' to 'debug' 2021-11-20T05:28:59.301 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 conmon[33537]: debug 2021-11-20T05:28:59.153+0000 7ff1bc8de700 20 expand_channel_meta expanded map: {default=debug} 2021-11-20T05:28:59.301 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 bash[33421]: cb3d93de393a230d69a6fb44d632c1932abb6cb30cb9c35819c8f38263301283 2021-11-20T05:28:59.302 INFO:teuthology.orchestra.run.smithi058.stderr:Setting mon public_network to 172.21.0.0/20 2021-11-20T05:28:59.302 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2021-11-20T05:28:59.502 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:28:59 smithi058 systemd[1]: Started Ceph mon.smithi058 for b33f8c4a-49c2-11ec-8c2c-001a4aab830c. 2021-11-20T05:28:59.786 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.782+0000 7f890dc20700 1 Processor -- start 2021-11-20T05:28:59.786 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.782+0000 7f890dc20700 1 -- start start 2021-11-20T05:28:59.786 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.782+0000 7f890dc20700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89080fc1f0 0x7f89080fc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:28:59.786 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.782+0000 7f890dc20700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89080fcb00 con 0x7f89080fc1f0 2021-11-20T05:28:59.787 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.782+0000 7f89077fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89080fc1f0 0x7f89080fc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:28:59.787 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.782+0000 7f89077fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89080fc1f0 0x7f89080fc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39506/0 (socket says 172.21.15.58:39506) 2021-11-20T05:28:59.787 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.782+0000 7f89077fe700 1 -- 172.21.15.58:0/3053893742 learned_addr learned my addr 172.21.15.58:0/3053893742 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:28:59.787 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.783+0000 7f89077fe700 1 -- 172.21.15.58:0/3053893742 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89080fcc40 con 0x7f89080fc1f0 2021-11-20T05:28:59.788 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.783+0000 7f89077fe700 1 --2- 172.21.15.58:0/3053893742 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89080fc1f0 0x7f89080fc610 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f88f800b5f0 tx=0x7f88f8006ee0).ready entity=mon.0 client_cookie=dfe8397b8e8e8802 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:28:59.788 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.783+0000 7f8906ffd700 1 -- 172.21.15.58:0/3053893742 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88f800d600 con 0x7f89080fc1f0 2021-11-20T05:28:59.788 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.783+0000 7f8906ffd700 1 -- 172.21.15.58:0/3053893742 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f88f800dbd0 con 0x7f89080fc1f0 2021-11-20T05:28:59.788 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.783+0000 7f8906ffd700 1 -- 172.21.15.58:0/3053893742 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88f8011c30 con 0x7f89080fc1f0 2021-11-20T05:28:59.789 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.784+0000 7f890dc20700 1 -- 172.21.15.58:0/3053893742 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89080fc1f0 msgr2=0x7f89080fc610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:28:59.789 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.784+0000 7f890dc20700 1 --2- 172.21.15.58:0/3053893742 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89080fc1f0 0x7f89080fc610 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f88f800b5f0 tx=0x7f88f8006ee0).stop 2021-11-20T05:28:59.789 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.784+0000 7f890dc20700 1 -- 172.21.15.58:0/3053893742 shutdown_connections 2021-11-20T05:28:59.789 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.784+0000 7f890dc20700 1 --2- 172.21.15.58:0/3053893742 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89080fc1f0 0x7f89080fc610 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:28:59.789 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.784+0000 7f890dc20700 1 -- 172.21.15.58:0/3053893742 >> 172.21.15.58:0/3053893742 conn(0x7f89080f5a80 msgr2=0x7f89080f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:28:59.790 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.784+0000 7f890dc20700 1 -- 172.21.15.58:0/3053893742 shutdown_connections 2021-11-20T05:28:59.790 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.784+0000 7f890dc20700 1 -- 172.21.15.58:0/3053893742 wait complete. 2021-11-20T05:28:59.790 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.784+0000 7f890dc20700 1 Processor -- start 2021-11-20T05:28:59.790 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.784+0000 7f890dc20700 1 -- start start 2021-11-20T05:28:59.791 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.785+0000 7f890dc20700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89080fc1f0 0x7f890810cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:28:59.791 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.785+0000 7f890dc20700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890810d0a0 con 0x7f89080fc1f0 2021-11-20T05:28:59.791 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.785+0000 7f89077fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89080fc1f0 0x7f890810cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:28:59.791 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.785+0000 7f89077fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89080fc1f0 0x7f890810cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39508/0 (socket says 172.21.15.58:39508) 2021-11-20T05:28:59.791 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.785+0000 7f89077fe700 1 -- 172.21.15.58:0/3320613589 learned_addr learned my addr 172.21.15.58:0/3320613589 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:28:59.792 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.785+0000 7f89077fe700 1 -- 172.21.15.58:0/3320613589 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f800b200 con 0x7f89080fc1f0 2021-11-20T05:28:59.792 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.785+0000 7f89077fe700 1 --2- 172.21.15.58:0/3320613589 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89080fc1f0 0x7f890810cbb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f88f800b1d0 tx=0x7f88f8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:28:59.792 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.785+0000 7f8904ff9700 1 -- 172.21.15.58:0/3320613589 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88f801b410 con 0x7f89080fc1f0 2021-11-20T05:28:59.792 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.785+0000 7f8904ff9700 1 -- 172.21.15.58:0/3320613589 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f88f801b9e0 con 0x7f89080fc1f0 2021-11-20T05:28:59.793 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.785+0000 7f890dc20700 1 -- 172.21.15.58:0/3320613589 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890810d2a0 con 0x7f89080fc1f0 2021-11-20T05:28:59.793 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.786+0000 7f8904ff9700 1 -- 172.21.15.58:0/3320613589 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88f801a7f0 con 0x7f89080fc1f0 2021-11-20T05:28:59.793 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.786+0000 7f890dc20700 1 -- 172.21.15.58:0/3320613589 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f890810d6b0 con 0x7f89080fc1f0 2021-11-20T05:28:59.793 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.786+0000 7f8904ff9700 1 -- 172.21.15.58:0/3320613589 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f88f800f070 con 0x7f89080fc1f0 2021-11-20T05:28:59.793 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.786+0000 7f8904ff9700 1 -- 172.21.15.58:0/3320613589 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f88f8025b30 con 0x7f89080fc1f0 2021-11-20T05:28:59.794 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.787+0000 7f890dc20700 1 -- 172.21.15.58:0/3320613589 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8908045ad0 con 0x7f89080fc1f0 2021-11-20T05:28:59.794 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.789+0000 7f8904ff9700 1 -- 172.21.15.58:0/3320613589 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f88f8016070 con 0x7f89080fc1f0 2021-11-20T05:28:59.908 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.909+0000 7f890dc20700 1 -- 172.21.15.58:0/3320613589 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f8908106aa0 con 0x7f89080fc1f0 2021-11-20T05:28:59.910 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.911+0000 7f8904ff9700 1 -- 172.21.15.58:0/3320613589 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f88f8025d70 con 0x7f89080fc1f0 2021-11-20T05:28:59.912 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.912+0000 7f890dc20700 1 -- 172.21.15.58:0/3320613589 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89080fc1f0 msgr2=0x7f890810cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:28:59.912 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.912+0000 7f890dc20700 1 --2- 172.21.15.58:0/3320613589 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89080fc1f0 0x7f890810cbb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f88f800b1d0 tx=0x7f88f8006ee0).stop 2021-11-20T05:28:59.912 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.912+0000 7f890dc20700 1 -- 172.21.15.58:0/3320613589 shutdown_connections 2021-11-20T05:28:59.912 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.912+0000 7f890dc20700 1 --2- 172.21.15.58:0/3320613589 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f89080fc1f0 0x7f890810cbb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:28:59.913 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.912+0000 7f890dc20700 1 -- 172.21.15.58:0/3320613589 >> 172.21.15.58:0/3320613589 conn(0x7f89080f5a80 msgr2=0x7f89080f75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:28:59.913 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.912+0000 7f890dc20700 1 -- 172.21.15.58:0/3320613589 shutdown_connections 2021-11-20T05:28:59.913 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:28:59.912+0000 7f890dc20700 1 -- 172.21.15.58:0/3320613589 wait complete. 2021-11-20T05:29:00.295 INFO:teuthology.orchestra.run.smithi058.stderr:Wrote config to /etc/ceph/ceph.conf 2021-11-20T05:29:00.295 INFO:teuthology.orchestra.run.smithi058.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2021-11-20T05:29:00.296 INFO:teuthology.orchestra.run.smithi058.stderr:Creating mgr... 2021-11-20T05:29:00.296 INFO:teuthology.orchestra.run.smithi058.stderr:Verifying port 9283 ... 2021-11-20T05:29:00.297 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl daemon-reload 2021-11-20T05:29:00.387 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:00 smithi058 conmon[33537]: cluster 2021-11-20T05:28:59.167420+0000 mon.smithi058 (mon.0) 1 : cluster [INF] mon.smithi058 is new leader, mons smithi058 in quorum (ranks 0) 2021-11-20T05:29:00.388 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:00 smithi058 conmon[33537]: cluster 2021-11-20T05:28:59.167472+0000 mon.smithi058 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi058=[v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0]} 2021-11-20T05:29:00.388 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:00 smithi058 conmon[33537]: cluster 2021-11-20T05:28:59.167504+0000 mon.smithi058 (mon.0) 3 : cluster [DBG] fsmap 2021-11-20T05:29:00.388 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:00 smithi058 conmon[33537]: cluster 2021-11-20T05:28:59.167531+0000 mon.smithi058 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-11-20T05:29:00.388 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:00 smithi058 conmon[33537]: cluster 2021-11-20T05:28:59.167870+0000 mon.smithi058 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2021-11-20T05:29:00.389 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:00 smithi058 conmon[33537]: audit 2021-11-20T05:28:59.912192+0000 mon.smithi058 (mon.0) 6 : audit [INF] from='client.? 172.21.15.58:0/3320613589' entity='client.admin' 2021-11-20T05:29:00.389 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl stop ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mgr.smithi058.zagdlk 2021-11-20T05:29:00.420 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl reset-failed ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mgr.smithi058.zagdlk 2021-11-20T05:29:00.425 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: Failed to reset failed state of unit ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mgr.smithi058.zagdlk.service: Unit ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mgr.smithi058.zagdlk.service not loaded. 2021-11-20T05:29:00.426 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl enable ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mgr.smithi058.zagdlk 2021-11-20T05:29:00.432 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: Created symlink /etc/systemd/system/ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c.target.wants/ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mgr.smithi058.zagdlk.service → /etc/systemd/system/ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@.service. 2021-11-20T05:29:00.515 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl start ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mgr.smithi058.zagdlk 2021-11-20T05:29:01.047 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-20T05:29:01.056 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: disabled 2021-11-20T05:29:01.056 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl is-active firewalld.service 2021-11-20T05:29:01.063 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: inactive 2021-11-20T05:29:01.065 INFO:teuthology.orchestra.run.smithi058.stderr:firewalld.service is not enabled 2021-11-20T05:29:01.066 INFO:teuthology.orchestra.run.smithi058.stderr:Not possible to enable service . firewalld.service is not available 2021-11-20T05:29:01.066 INFO:teuthology.orchestra.run.smithi058.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-11-20T05:29:01.066 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-20T05:29:01.071 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: disabled 2021-11-20T05:29:01.072 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl is-active firewalld.service 2021-11-20T05:29:01.079 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: inactive 2021-11-20T05:29:01.080 INFO:teuthology.orchestra.run.smithi058.stderr:firewalld.service is not enabled 2021-11-20T05:29:01.080 INFO:teuthology.orchestra.run.smithi058.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2021-11-20T05:29:01.081 INFO:teuthology.orchestra.run.smithi058.stderr:Waiting for mgr to start... 2021-11-20T05:29:01.081 INFO:teuthology.orchestra.run.smithi058.stderr:Waiting for mgr... 2021-11-20T05:29:01.081 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2021-11-20T05:29:01.583 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.579+0000 7f40062cd700 1 Processor -- start 2021-11-20T05:29:01.583 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.579+0000 7f40062cd700 1 -- start start 2021-11-20T05:29:01.584 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.579+0000 7f40062cd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f400006fcd0 0x7f40000fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:01.584 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.579+0000 7f40062cd700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40000fed20 con 0x7f400006fcd0 2021-11-20T05:29:01.584 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.579+0000 7f3ffffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f400006fcd0 0x7f40000fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:01.584 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.579+0000 7f3ffffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f400006fcd0 0x7f40000fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39514/0 (socket says 172.21.15.58:39514) 2021-11-20T05:29:01.585 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.579+0000 7f3ffffff700 1 -- 172.21.15.58:0/2885989758 learned_addr learned my addr 172.21.15.58:0/2885989758 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:01.585 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.580+0000 7f3ffffff700 1 -- 172.21.15.58:0/2885989758 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40000fee60 con 0x7f400006fcd0 2021-11-20T05:29:01.585 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.580+0000 7f3ffffff700 1 --2- 172.21.15.58:0/2885989758 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f400006fcd0 0x7f40000fe7a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f3ff000b5f0 tx=0x7f3ff0006ee0).ready entity=mon.0 client_cookie=8c95006b624b7d52 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:01.585 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.580+0000 7f3ffeffd700 1 -- 172.21.15.58:0/2885989758 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ff000d600 con 0x7f400006fcd0 2021-11-20T05:29:01.586 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.580+0000 7f3ffeffd700 1 -- 172.21.15.58:0/2885989758 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3ff000dbd0 con 0x7f400006fcd0 2021-11-20T05:29:01.586 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.580+0000 7f3ffeffd700 1 -- 172.21.15.58:0/2885989758 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ff0011c00 con 0x7f400006fcd0 2021-11-20T05:29:01.586 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.581+0000 7f40062cd700 1 -- 172.21.15.58:0/2885989758 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f400006fcd0 msgr2=0x7f40000fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:01.586 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.581+0000 7f40062cd700 1 --2- 172.21.15.58:0/2885989758 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f400006fcd0 0x7f40000fe7a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f3ff000b5f0 tx=0x7f3ff0006ee0).stop 2021-11-20T05:29:01.587 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.581+0000 7f40062cd700 1 -- 172.21.15.58:0/2885989758 shutdown_connections 2021-11-20T05:29:01.587 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.581+0000 7f40062cd700 1 --2- 172.21.15.58:0/2885989758 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f400006fcd0 0x7f40000fe7a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:01.587 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.581+0000 7f40062cd700 1 -- 172.21.15.58:0/2885989758 >> 172.21.15.58:0/2885989758 conn(0x7f40000f6280 msgr2=0x7f40000f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:01.587 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.581+0000 7f40062cd700 1 -- 172.21.15.58:0/2885989758 shutdown_connections 2021-11-20T05:29:01.587 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.581+0000 7f40062cd700 1 -- 172.21.15.58:0/2885989758 wait complete. 2021-11-20T05:29:01.588 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.582+0000 7f40062cd700 1 Processor -- start 2021-11-20T05:29:01.588 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.582+0000 7f40062cd700 1 -- start start 2021-11-20T05:29:01.588 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.582+0000 7f40062cd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f400006fcd0 0x7f400010cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:01.588 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.582+0000 7f40062cd700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400010d0d0 con 0x7f400006fcd0 2021-11-20T05:29:01.589 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.582+0000 7f3ffffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f400006fcd0 0x7f400010cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:01.589 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.582+0000 7f3ffffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f400006fcd0 0x7f400010cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39516/0 (socket says 172.21.15.58:39516) 2021-11-20T05:29:01.589 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.582+0000 7f3ffffff700 1 -- 172.21.15.58:0/931890509 learned_addr learned my addr 172.21.15.58:0/931890509 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:01.589 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.583+0000 7f3ffffff700 1 -- 172.21.15.58:0/931890509 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff000b200 con 0x7f400006fcd0 2021-11-20T05:29:01.589 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.583+0000 7f3ffffff700 1 --2- 172.21.15.58:0/931890509 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f400006fcd0 0x7f400010cbe0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f3ff0000c00 tx=0x7f3ff0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:01.590 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.583+0000 7f3ffcff9700 1 -- 172.21.15.58:0/931890509 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ff0011690 con 0x7f400006fcd0 2021-11-20T05:29:01.590 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.583+0000 7f3ffcff9700 1 -- 172.21.15.58:0/931890509 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3ff000d9f0 con 0x7f400006fcd0 2021-11-20T05:29:01.591 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.583+0000 7f40062cd700 1 -- 172.21.15.58:0/931890509 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f400010d2d0 con 0x7f400006fcd0 2021-11-20T05:29:01.591 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.583+0000 7f3ffcff9700 1 -- 172.21.15.58:0/931890509 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ff001b710 con 0x7f400006fcd0 2021-11-20T05:29:01.591 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.583+0000 7f40062cd700 1 -- 172.21.15.58:0/931890509 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f400010d6e0 con 0x7f400006fcd0 2021-11-20T05:29:01.591 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.583+0000 7f3ffcff9700 1 -- 172.21.15.58:0/931890509 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f3ff000f070 con 0x7f400006fcd0 2021-11-20T05:29:01.592 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.583+0000 7f3ffcff9700 1 -- 172.21.15.58:0/931890509 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f3ff0025b30 con 0x7f400006fcd0 2021-11-20T05:29:01.592 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.584+0000 7f40062cd700 1 -- 172.21.15.58:0/931890509 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4000045ad0 con 0x7f400006fcd0 2021-11-20T05:29:01.593 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.586+0000 7f3ffcff9700 1 -- 172.21.15.58:0/931890509 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f3ff0016070 con 0x7f400006fcd0 2021-11-20T05:29:01.718 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.711+0000 7f40062cd700 1 -- 172.21.15.58:0/931890509 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f40000fb1e0 con 0x7f400006fcd0 2021-11-20T05:29:01.718 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.712+0000 7f3ffcff9700 1 -- 172.21.15.58:0/931890509 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f3ff001bd00 con 0x7f400006fcd0 2021-11-20T05:29:01.719 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.719 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: { 2021-11-20T05:29:01.719 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "fsid": "b33f8c4a-49c2-11ec-8c2c-001a4aab830c", 2021-11-20T05:29:01.719 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "health": { 2021-11-20T05:29:01.719 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-11-20T05:29:01.720 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "checks": {}, 2021-11-20T05:29:01.720 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "mutes": [] 2021-11-20T05:29:01.720 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:01.721 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-11-20T05:29:01.721 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "quorum": [ 2021-11-20T05:29:01.721 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 0 2021-11-20T05:29:01.721 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: ], 2021-11-20T05:29:01.722 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "quorum_names": [ 2021-11-20T05:29:01.722 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "smithi058" 2021-11-20T05:29:01.722 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: ], 2021-11-20T05:29:01.722 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "quorum_age": 2, 2021-11-20T05:29:01.723 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "monmap": { 2021-11-20T05:29:01.723 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:29:01.723 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-11-20T05:29:01.723 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_mons": 1 2021-11-20T05:29:01.724 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:01.724 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "osdmap": { 2021-11-20T05:29:01.724 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:29:01.724 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_osds": 0, 2021-11-20T05:29:01.724 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-11-20T05:29:01.725 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-11-20T05:29:01.725 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-11-20T05:29:01.725 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-11-20T05:29:01.725 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-11-20T05:29:01.726 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:01.726 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "pgmap": { 2021-11-20T05:29:01.726 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-11-20T05:29:01.727 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-11-20T05:29:01.727 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_pools": 0, 2021-11-20T05:29:01.727 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_objects": 0, 2021-11-20T05:29:01.727 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-11-20T05:29:01.728 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-11-20T05:29:01.728 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-11-20T05:29:01.728 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "bytes_total": 0 2021-11-20T05:29:01.729 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:01.729 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "fsmap": { 2021-11-20T05:29:01.729 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:29:01.729 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "by_rank": [], 2021-11-20T05:29:01.729 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "up:standby": 0 2021-11-20T05:29:01.730 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:01.730 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "mgrmap": { 2021-11-20T05:29:01.730 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "available": false, 2021-11-20T05:29:01.730 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-11-20T05:29:01.731 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "modules": [ 2021-11-20T05:29:01.731 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "iostat", 2021-11-20T05:29:01.731 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "restful" 2021-11-20T05:29:01.733 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: ], 2021-11-20T05:29:01.733 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "services": {} 2021-11-20T05:29:01.733 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:01.734 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "servicemap": { 2021-11-20T05:29:01.734 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:29:01.734 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "modified": "2021-11-20T05:28:54.787821+0000", 2021-11-20T05:29:01.735 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "services": {} 2021-11-20T05:29:01.735 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:01.735 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "progress_events": {} 2021-11-20T05:29:01.735 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: } 2021-11-20T05:29:01.735 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.713+0000 7f40062cd700 1 -- 172.21.15.58:0/931890509 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f400006fcd0 msgr2=0x7f400010cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:01.736 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.713+0000 7f40062cd700 1 --2- 172.21.15.58:0/931890509 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f400006fcd0 0x7f400010cbe0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f3ff0000c00 tx=0x7f3ff0006ee0).stop 2021-11-20T05:29:01.736 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.713+0000 7f40062cd700 1 -- 172.21.15.58:0/931890509 shutdown_connections 2021-11-20T05:29:01.736 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.713+0000 7f40062cd700 1 --2- 172.21.15.58:0/931890509 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f400006fcd0 0x7f400010cbe0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:01.736 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.713+0000 7f40062cd700 1 -- 172.21.15.58:0/931890509 >> 172.21.15.58:0/931890509 conn(0x7f40000f6280 msgr2=0x7f40000fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:01.737 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.713+0000 7f40062cd700 1 -- 172.21.15.58:0/931890509 shutdown_connections 2021-11-20T05:29:01.737 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:01.713+0000 7f40062cd700 1 -- 172.21.15.58:0/931890509 wait complete. 2021-11-20T05:29:01.888 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:01 smithi058 conmon[33537]: audit 2021-11-20T05:29:01.712955+0000 mon.smithi058 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.58:0/931890509' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-11-20T05:29:02.121 INFO:teuthology.orchestra.run.smithi058.stderr:mgr not available, waiting (1/15)... 2021-11-20T05:29:04.124 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2021-11-20T05:29:04.642 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.639+0000 7f9b2eb70700 1 Processor -- start 2021-11-20T05:29:04.643 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.639+0000 7f9b2eb70700 1 -- start start 2021-11-20T05:29:04.643 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.639+0000 7f9b2eb70700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b28071c90 0x7f9b28070230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:04.643 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.639+0000 7f9b2eb70700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b280720b0 con 0x7f9b28071c90 2021-11-20T05:29:04.644 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.640+0000 7f9b2c90c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b28071c90 0x7f9b28070230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:04.644 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.640+0000 7f9b2c90c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b28071c90 0x7f9b28070230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39518/0 (socket says 172.21.15.58:39518) 2021-11-20T05:29:04.644 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.640+0000 7f9b2c90c700 1 -- 172.21.15.58:0/2884440924 learned_addr learned my addr 172.21.15.58:0/2884440924 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:04.644 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.640+0000 7f9b2c90c700 1 -- 172.21.15.58:0/2884440924 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b28070720 con 0x7f9b28071c90 2021-11-20T05:29:04.645 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.640+0000 7f9b2c90c700 1 --2- 172.21.15.58:0/2884440924 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b28071c90 0x7f9b28070230 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f9b1c00b5f0 tx=0x7f9b1c006ee0).ready entity=mon.0 client_cookie=167b6874262027e4 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:04.645 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.640+0000 7f9b277fe700 1 -- 172.21.15.58:0/2884440924 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b1c00d600 con 0x7f9b28071c90 2021-11-20T05:29:04.645 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.640+0000 7f9b277fe700 1 -- 172.21.15.58:0/2884440924 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9b1c00dbd0 con 0x7f9b28071c90 2021-11-20T05:29:04.645 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.640+0000 7f9b277fe700 1 -- 172.21.15.58:0/2884440924 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b1c011c00 con 0x7f9b28071c90 2021-11-20T05:29:04.646 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.641+0000 7f9b2eb70700 1 -- 172.21.15.58:0/2884440924 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b28071c90 msgr2=0x7f9b28070230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:04.646 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.641+0000 7f9b2eb70700 1 --2- 172.21.15.58:0/2884440924 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b28071c90 0x7f9b28070230 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f9b1c00b5f0 tx=0x7f9b1c006ee0).stop 2021-11-20T05:29:04.646 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.641+0000 7f9b2eb70700 1 -- 172.21.15.58:0/2884440924 shutdown_connections 2021-11-20T05:29:04.646 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.641+0000 7f9b2eb70700 1 --2- 172.21.15.58:0/2884440924 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b28071c90 0x7f9b28070230 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:04.647 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.641+0000 7f9b2eb70700 1 -- 172.21.15.58:0/2884440924 >> 172.21.15.58:0/2884440924 conn(0x7f9b280f73f0 msgr2=0x7f9b280f9810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:04.647 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.641+0000 7f9b2eb70700 1 -- 172.21.15.58:0/2884440924 shutdown_connections 2021-11-20T05:29:04.647 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.641+0000 7f9b2eb70700 1 -- 172.21.15.58:0/2884440924 wait complete. 2021-11-20T05:29:04.647 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.642+0000 7f9b2eb70700 1 Processor -- start 2021-11-20T05:29:04.647 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.642+0000 7f9b2eb70700 1 -- start start 2021-11-20T05:29:04.648 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.642+0000 7f9b2eb70700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b28071c90 0x7f9b2810cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:04.648 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.642+0000 7f9b2eb70700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2810d080 con 0x7f9b28071c90 2021-11-20T05:29:04.648 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.643+0000 7f9b2c90c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b28071c90 0x7f9b2810cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:04.648 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.643+0000 7f9b2c90c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b28071c90 0x7f9b2810cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39520/0 (socket says 172.21.15.58:39520) 2021-11-20T05:29:04.649 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.643+0000 7f9b2c90c700 1 -- 172.21.15.58:0/2061118400 learned_addr learned my addr 172.21.15.58:0/2061118400 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:04.650 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.643+0000 7f9b2c90c700 1 -- 172.21.15.58:0/2061118400 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1c00b200 con 0x7f9b28071c90 2021-11-20T05:29:04.650 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.643+0000 7f9b2c90c700 1 --2- 172.21.15.58:0/2061118400 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b28071c90 0x7f9b2810cb90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f9b1c011820 tx=0x7f9b1c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:04.651 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.643+0000 7f9b257fa700 1 -- 172.21.15.58:0/2061118400 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b1c01b410 con 0x7f9b28071c90 2021-11-20T05:29:04.651 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.643+0000 7f9b257fa700 1 -- 172.21.15.58:0/2061118400 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9b1c01b9e0 con 0x7f9b28071c90 2021-11-20T05:29:04.651 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.643+0000 7f9b2eb70700 1 -- 172.21.15.58:0/2061118400 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b2810d280 con 0x7f9b28071c90 2021-11-20T05:29:04.651 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.643+0000 7f9b257fa700 1 -- 172.21.15.58:0/2061118400 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b1c01a8b0 con 0x7f9b28071c90 2021-11-20T05:29:04.652 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.643+0000 7f9b2eb70700 1 -- 172.21.15.58:0/2061118400 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b2810d690 con 0x7f9b28071c90 2021-11-20T05:29:04.652 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.643+0000 7f9b257fa700 1 -- 172.21.15.58:0/2061118400 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f9b1c00f070 con 0x7f9b28071c90 2021-11-20T05:29:04.652 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.644+0000 7f9b257fa700 1 -- 172.21.15.58:0/2061118400 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9b1c025b80 con 0x7f9b28071c90 2021-11-20T05:29:04.652 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.645+0000 7f9b2eb70700 1 -- 172.21.15.58:0/2061118400 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b08004fa0 con 0x7f9b28071c90 2021-11-20T05:29:04.652 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.647+0000 7f9b257fa700 1 -- 172.21.15.58:0/2061118400 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9b1c016070 con 0x7f9b28071c90 2021-11-20T05:29:04.778 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.771+0000 7f9b2eb70700 1 -- 172.21.15.58:0/2061118400 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9b08005670 con 0x7f9b28071c90 2021-11-20T05:29:04.778 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.772+0000 7f9b257fa700 1 -- 172.21.15.58:0/2061118400 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f9b1c025d90 con 0x7f9b28071c90 2021-11-20T05:29:04.779 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.779 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: { 2021-11-20T05:29:04.779 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "fsid": "b33f8c4a-49c2-11ec-8c2c-001a4aab830c", 2021-11-20T05:29:04.779 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "health": { 2021-11-20T05:29:04.780 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-11-20T05:29:04.780 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "checks": {}, 2021-11-20T05:29:04.780 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "mutes": [] 2021-11-20T05:29:04.780 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:04.781 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-11-20T05:29:04.781 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "quorum": [ 2021-11-20T05:29:04.781 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 0 2021-11-20T05:29:04.781 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: ], 2021-11-20T05:29:04.782 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "quorum_names": [ 2021-11-20T05:29:04.782 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "smithi058" 2021-11-20T05:29:04.782 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: ], 2021-11-20T05:29:04.782 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "quorum_age": 5, 2021-11-20T05:29:04.782 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "monmap": { 2021-11-20T05:29:04.783 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:29:04.783 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-11-20T05:29:04.783 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_mons": 1 2021-11-20T05:29:04.783 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:04.784 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "osdmap": { 2021-11-20T05:29:04.784 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:29:04.784 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_osds": 0, 2021-11-20T05:29:04.785 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-11-20T05:29:04.785 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-11-20T05:29:04.785 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-11-20T05:29:04.785 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-11-20T05:29:04.786 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-11-20T05:29:04.786 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:04.786 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "pgmap": { 2021-11-20T05:29:04.786 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-11-20T05:29:04.787 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-11-20T05:29:04.787 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_pools": 0, 2021-11-20T05:29:04.787 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_objects": 0, 2021-11-20T05:29:04.787 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-11-20T05:29:04.788 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-11-20T05:29:04.788 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-11-20T05:29:04.788 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "bytes_total": 0 2021-11-20T05:29:04.788 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:04.788 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "fsmap": { 2021-11-20T05:29:04.789 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:29:04.789 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "by_rank": [], 2021-11-20T05:29:04.789 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "up:standby": 0 2021-11-20T05:29:04.789 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:04.790 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "mgrmap": { 2021-11-20T05:29:04.790 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "available": false, 2021-11-20T05:29:04.790 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-11-20T05:29:04.790 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "modules": [ 2021-11-20T05:29:04.791 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "iostat", 2021-11-20T05:29:04.791 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "restful" 2021-11-20T05:29:04.791 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: ], 2021-11-20T05:29:04.791 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "services": {} 2021-11-20T05:29:04.792 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:04.792 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "servicemap": { 2021-11-20T05:29:04.792 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:29:04.792 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "modified": "2021-11-20T05:28:54.787821+0000", 2021-11-20T05:29:04.793 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "services": {} 2021-11-20T05:29:04.793 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:04.793 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "progress_events": {} 2021-11-20T05:29:04.794 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: } 2021-11-20T05:29:04.794 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.773+0000 7f9b2eb70700 1 -- 172.21.15.58:0/2061118400 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b28071c90 msgr2=0x7f9b2810cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:04.794 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.773+0000 7f9b2eb70700 1 --2- 172.21.15.58:0/2061118400 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b28071c90 0x7f9b2810cb90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f9b1c011820 tx=0x7f9b1c006ee0).stop 2021-11-20T05:29:04.794 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.773+0000 7f9b2eb70700 1 -- 172.21.15.58:0/2061118400 shutdown_connections 2021-11-20T05:29:04.795 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.773+0000 7f9b2eb70700 1 --2- 172.21.15.58:0/2061118400 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b28071c90 0x7f9b2810cb90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:04.795 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.773+0000 7f9b2eb70700 1 -- 172.21.15.58:0/2061118400 >> 172.21.15.58:0/2061118400 conn(0x7f9b280f73f0 msgr2=0x7f9b280f8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:04.795 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.773+0000 7f9b2eb70700 1 -- 172.21.15.58:0/2061118400 shutdown_connections 2021-11-20T05:29:04.795 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:04.773+0000 7f9b2eb70700 1 -- 172.21.15.58:0/2061118400 wait complete. 2021-11-20T05:29:05.162 INFO:teuthology.orchestra.run.smithi058.stderr:mgr not available, waiting (2/15)... 2021-11-20T05:29:05.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:05 smithi058 conmon[33537]: audit 2021-11-20T05:29:04.773077+0000 2021-11-20T05:29:05.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:05 smithi058 conmon[33537]: mon.smithi058 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.58:0/2061118400' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-11-20T05:29:07.164 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2021-11-20T05:29:07.385 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: cluster 2021-11-20T05:29:07.013900+0000 mon.smithi058 2021-11-20T05:29:07.385 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: (mon.0) 9 : cluster [INF] Activating manager daemon smithi058.zagdlk 2021-11-20T05:29:07.386 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: cluster 2021-11-20T05:29:07.016447+0000 mon.smithi058 (mon.0) 10 : cluster [DBG] 2021-11-20T05:29:07.386 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: mgrmap e2: smithi058.zagdlk(active, starting, since 0.00265623s) 2021-11-20T05:29:07.386 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: audit 2021-11- 2021-11-20T05:29:07.387 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: 20T05:29:07.017597+0000 mon.smithi058 (mon. 2021-11-20T05:29:07.387 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: 0) 11 : audit [DBG] from='mgr.14100 172.21.15.58:0/405652257' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-20T05:29:07.387 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:29:07.388 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: -20T05:29:07.017778+0000 2021-11-20T05:29:07.388 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: mon.smithi058 (mon.0) 12 : audit [DBG] 2021-11-20T05:29:07.389 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: from='mgr.14100 172.21.15.58:0/405652257' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-20T05:29:07.389 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: audit 2021-11-20T05:29:07.389 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: 2021-11-20T05:29:07.017920+0000 2021-11-20T05:29:07.390 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: mon.smithi058 (mon.0) 13 : audit [DBG] 2021-11-20T05:29:07.390 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: from='mgr.14100 172.21.15.58:0/405652257' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-20T05:29:07.391 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: audit 2021-11-20T05:29:07.391 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: 2021-11-20T05:29:07. 2021-11-20T05:29:07.392 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: 018089+0000 mon.smithi058 (mon.0) 14 : audit 2021-11-20T05:29:07.392 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: [DBG] from='mgr.14100 172.21.15.58:0/405652257' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi058"}]: dispatch 2021-11-20T05:29:07.393 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: audit 2021-11-20T05 2021-11-20T05:29:07.394 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: :29:07.018274+0000 mon.smithi058 (mon. 2021-11-20T05:29:07.394 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: 0) 15 : audit [DBG] from='mgr.14100 172.21.15.58:0/405652257' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mgr metadata", "who": "smithi058.zagdlk", "id": "smithi058.zagdlk"}]: dispatch 2021-11-20T05:29:07.394 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: cluster 2021- 2021-11-20T05:29:07.395 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: 11-20T05:29:07.032473+0000 2021-11-20T05:29:07.395 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: mon.smithi058 (mon.0) 16 : cluster [INF] 2021-11-20T05:29:07.397 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: Manager daemon smithi058.zagdlk is now available 2021-11-20T05:29:07.398 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: audit 2021-11-20T05:29:07.398 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: 2021-11-20T05:29:07. 2021-11-20T05:29:07.398 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: 041062+0000 mon.smithi058 (mon.0) 17 : audit 2021-11-20T05:29:07.399 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: [INF] from='mgr.14100 172.21.15.58:0/405652257' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:07.399 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: audit 2021-11-20T05 2021-11-20T05:29:07.400 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: :29:07.045364+0000 mon.smithi058 (mon 2021-11-20T05:29:07.400 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: .0) 18 : audit [INF] from='mgr.14100 172.21.15.58:0/405652257' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:29:07.401 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:29:07.401 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: -20T05:29:07.046577+0000 mon.smithi058 2021-11-20T05:29:07.402 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: (mon.0) 19 : audit [INF] 2021-11-20T05:29:07.403 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: from='mgr.14100 172.21.15.58:0/405652257' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:29:07.403 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: audit 2021-11-20T05:29:07.403 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: 2021-11-20T05:29:07.051797+0000 2021-11-20T05:29:07.404 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: mon.smithi058 (mon.0) 20 : audit [INF] 2021-11-20T05:29:07.404 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: from='mgr.14100 172.21.15.58:0/405652257' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:07.405 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: audit 2021-11-20T05:29:07.405 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: 2021-11-20T05:29:07. 2021-11-20T05:29:07.407 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: 053718+0000 mon.smithi058 (mon.0) 21 : audit 2021-11-20T05:29:07.407 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:07 smithi058 conmon[33537]: [INF] from='mgr.14100 172.21.15.58:0/405652257' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:07.644 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.640+0000 7f43ae052700 1 Processor -- start 2021-11-20T05:29:07.644 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.640+0000 7f43ae052700 1 -- start start 2021-11-20T05:29:07.644 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.641+0000 7f43ae052700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f43a806fc90 0x7f43a80fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:07.645 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.641+0000 7f43ae052700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a80fed60 con 0x7f43a806fc90 2021-11-20T05:29:07.645 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.641+0000 7f43a77fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f43a806fc90 0x7f43a80fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:07.645 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.641+0000 7f43a77fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f43a806fc90 0x7f43a80fe7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39532/0 (socket says 172.21.15.58:39532) 2021-11-20T05:29:07.645 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.641+0000 7f43a77fe700 1 -- 172.21.15.58:0/1257034657 learned_addr learned my addr 172.21.15.58:0/1257034657 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:07.646 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.641+0000 7f43a77fe700 1 -- 172.21.15.58:0/1257034657 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a80feea0 con 0x7f43a806fc90 2021-11-20T05:29:07.646 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.641+0000 7f43a77fe700 1 --2- 172.21.15.58:0/1257034657 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f43a806fc90 0x7f43a80fe7e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f43a0009960 tx=0x7f43a00092d0).ready entity=mon.0 client_cookie=bd91b54ddbe82230 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:07.646 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.641+0000 7f43a67fc700 1 -- 172.21.15.58:0/1257034657 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43a000b9c0 con 0x7f43a806fc90 2021-11-20T05:29:07.648 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.641+0000 7f43a67fc700 1 -- 172.21.15.58:0/1257034657 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f43a0011400 con 0x7f43a806fc90 2021-11-20T05:29:07.648 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.642+0000 7f43a67fc700 1 -- 172.21.15.58:0/1257034657 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43a00103a0 con 0x7f43a806fc90 2021-11-20T05:29:07.650 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.642+0000 7f43ae052700 1 -- 172.21.15.58:0/1257034657 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f43a806fc90 msgr2=0x7f43a80fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:07.650 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.642+0000 7f43ae052700 1 --2- 172.21.15.58:0/1257034657 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f43a806fc90 0x7f43a80fe7e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f43a0009960 tx=0x7f43a00092d0).stop 2021-11-20T05:29:07.650 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.642+0000 7f43ae052700 1 -- 172.21.15.58:0/1257034657 shutdown_connections 2021-11-20T05:29:07.650 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.642+0000 7f43ae052700 1 --2- 172.21.15.58:0/1257034657 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f43a806fc90 0x7f43a80fe7e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:07.651 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.642+0000 7f43ae052700 1 -- 172.21.15.58:0/1257034657 >> 172.21.15.58:0/1257034657 conn(0x7f43a80f6280 msgr2=0x7f43a80f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:07.651 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.642+0000 7f43ae052700 1 -- 172.21.15.58:0/1257034657 shutdown_connections 2021-11-20T05:29:07.651 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.642+0000 7f43ae052700 1 -- 172.21.15.58:0/1257034657 wait complete. 2021-11-20T05:29:07.651 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.643+0000 7f43ae052700 1 Processor -- start 2021-11-20T05:29:07.651 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.643+0000 7f43ae052700 1 -- start start 2021-11-20T05:29:07.652 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.643+0000 7f43ae052700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f43a806fc90 0x7f43a810a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:07.652 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.643+0000 7f43ae052700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a810ae90 con 0x7f43a806fc90 2021-11-20T05:29:07.652 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.643+0000 7f43a77fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f43a806fc90 0x7f43a810a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:07.652 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.643+0000 7f43a77fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f43a806fc90 0x7f43a810a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39534/0 (socket says 172.21.15.58:39534) 2021-11-20T05:29:07.653 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.643+0000 7f43a77fe700 1 -- 172.21.15.58:0/2651673018 learned_addr learned my addr 172.21.15.58:0/2651673018 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:07.653 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.644+0000 7f43a77fe700 1 -- 172.21.15.58:0/2651673018 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a0009570 con 0x7f43a806fc90 2021-11-20T05:29:07.653 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.644+0000 7f43a77fe700 1 --2- 172.21.15.58:0/2651673018 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f43a806fc90 0x7f43a810a9a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f43a0000f80 tx=0x7f43a00092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:07.653 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.644+0000 7f439ffff700 1 -- 172.21.15.58:0/2651673018 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43a0010820 con 0x7f43a806fc90 2021-11-20T05:29:07.653 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.644+0000 7f439ffff700 1 -- 172.21.15.58:0/2651673018 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f43a0010df0 con 0x7f43a806fc90 2021-11-20T05:29:07.654 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.644+0000 7f439ffff700 1 -- 172.21.15.58:0/2651673018 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43a001ae40 con 0x7f43a806fc90 2021-11-20T05:29:07.654 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.644+0000 7f43ae052700 1 -- 172.21.15.58:0/2651673018 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a810b090 con 0x7f43a806fc90 2021-11-20T05:29:07.654 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.644+0000 7f43ae052700 1 -- 172.21.15.58:0/2651673018 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a810b4a0 con 0x7f43a806fc90 2021-11-20T05:29:07.655 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.645+0000 7f439ffff700 1 -- 172.21.15.58:0/2651673018 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 2) v1 ==== 35959+0+0 (secure 0 0 0) 0x7f43a000f070 con 0x7f43a806fc90 2021-11-20T05:29:07.655 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.645+0000 7f43ae052700 1 -- 172.21.15.58:0/2651673018 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a8045ad0 con 0x7f43a806fc90 2021-11-20T05:29:07.655 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.645+0000 7f439ffff700 1 -- 172.21.15.58:0/2651673018 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f43a0041c80 con 0x7f43a806fc90 2021-11-20T05:29:07.656 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.647+0000 7f439ffff700 1 -- 172.21.15.58:0/2651673018 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f43a0018320 con 0x7f43a806fc90 2021-11-20T05:29:07.779 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.772+0000 7f43ae052700 1 -- 172.21.15.58:0/2651673018 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f43a80ffbd0 con 0x7f43a806fc90 2021-11-20T05:29:07.779 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.772+0000 7f439ffff700 1 -- 172.21.15.58:0/2651673018 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f43a001e070 con 0x7f43a806fc90 2021-11-20T05:29:07.779 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.780 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: { 2021-11-20T05:29:07.780 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "fsid": "b33f8c4a-49c2-11ec-8c2c-001a4aab830c", 2021-11-20T05:29:07.780 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "health": { 2021-11-20T05:29:07.780 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-11-20T05:29:07.781 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "checks": {}, 2021-11-20T05:29:07.781 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "mutes": [] 2021-11-20T05:29:07.781 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:07.781 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-11-20T05:29:07.781 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "quorum": [ 2021-11-20T05:29:07.782 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 0 2021-11-20T05:29:07.782 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: ], 2021-11-20T05:29:07.782 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "quorum_names": [ 2021-11-20T05:29:07.782 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "smithi058" 2021-11-20T05:29:07.783 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: ], 2021-11-20T05:29:07.783 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "quorum_age": 8, 2021-11-20T05:29:07.783 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "monmap": { 2021-11-20T05:29:07.783 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:29:07.784 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-11-20T05:29:07.784 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_mons": 1 2021-11-20T05:29:07.784 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:07.784 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "osdmap": { 2021-11-20T05:29:07.784 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:29:07.785 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_osds": 0, 2021-11-20T05:29:07.785 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-11-20T05:29:07.785 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-11-20T05:29:07.785 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-11-20T05:29:07.786 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-11-20T05:29:07.786 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-11-20T05:29:07.786 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:07.786 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "pgmap": { 2021-11-20T05:29:07.786 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-11-20T05:29:07.787 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-11-20T05:29:07.787 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_pools": 0, 2021-11-20T05:29:07.787 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_objects": 0, 2021-11-20T05:29:07.787 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-11-20T05:29:07.788 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-11-20T05:29:07.788 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-11-20T05:29:07.788 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "bytes_total": 0 2021-11-20T05:29:07.788 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:07.788 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "fsmap": { 2021-11-20T05:29:07.789 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:29:07.789 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "by_rank": [], 2021-11-20T05:29:07.789 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "up:standby": 0 2021-11-20T05:29:07.789 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:07.790 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "mgrmap": { 2021-11-20T05:29:07.790 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "available": false, 2021-11-20T05:29:07.790 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-11-20T05:29:07.790 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "modules": [ 2021-11-20T05:29:07.790 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "iostat", 2021-11-20T05:29:07.791 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "restful" 2021-11-20T05:29:07.791 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: ], 2021-11-20T05:29:07.791 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "services": {} 2021-11-20T05:29:07.791 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:07.792 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "servicemap": { 2021-11-20T05:29:07.792 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:29:07.792 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "modified": "2021-11-20T05:28:54.787821+0000", 2021-11-20T05:29:07.792 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "services": {} 2021-11-20T05:29:07.793 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:07.793 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "progress_events": {} 2021-11-20T05:29:07.793 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: } 2021-11-20T05:29:07.793 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.773+0000 7f43ae052700 1 -- 172.21.15.58:0/2651673018 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f43a806fc90 msgr2=0x7f43a810a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:07.793 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.773+0000 7f43ae052700 1 --2- 172.21.15.58:0/2651673018 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f43a806fc90 0x7f43a810a9a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f43a0000f80 tx=0x7f43a00092d0).stop 2021-11-20T05:29:07.794 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.774+0000 7f43ae052700 1 -- 172.21.15.58:0/2651673018 shutdown_connections 2021-11-20T05:29:07.794 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.774+0000 7f43ae052700 1 --2- 172.21.15.58:0/2651673018 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f43a806fc90 0x7f43a810a9a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:07.794 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.774+0000 7f43ae052700 1 -- 172.21.15.58:0/2651673018 >> 172.21.15.58:0/2651673018 conn(0x7f43a80f6280 msgr2=0x7f43a80f7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:07.794 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.774+0000 7f43ae052700 1 -- 172.21.15.58:0/2651673018 shutdown_connections 2021-11-20T05:29:07.795 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:07.774+0000 7f43ae052700 1 -- 172.21.15.58:0/2651673018 wait complete. 2021-11-20T05:29:08.099 INFO:teuthology.orchestra.run.smithi058.stderr:mgr not available, waiting (3/15)... 2021-11-20T05:29:08.219 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:08 smithi058 conmon[33537]: audit 2021-11-20T05:29:07.773542+0000 mon.smithi058 (mon.0 2021-11-20T05:29:08.219 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:08 smithi058 conmon[33537]: ) 22 : audit [DBG] from='client.? 172.21.15.58:0/2651673018' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-11-20T05:29:08.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:08 smithi058 conmon[33537]: cluster 2021-11-20T05:29:08.018869+0000 mon.smithi058 (mon.0) 23 : cluster [DBG] mgrmap e3: smithi058.zagdlk(active, since 1.00508s) 2021-11-20T05:29:10.101 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2021-11-20T05:29:10.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:10 smithi058 conmon[33537]: cluster 2021-11-20T05:29:09.069074+0000 mon.smithi058 (mon.0) 24 : 2021-11-20T05:29:10.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:10 smithi058 conmon[33537]: cluster [DBG] mgrmap e4: smithi058.zagdlk(active, since 2s) 2021-11-20T05:29:11.608 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.600+0000 7f83afff2700 1 Processor -- start 2021-11-20T05:29:11.608 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.600+0000 7f83afff2700 1 -- start start 2021-11-20T05:29:11.608 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.600+0000 7f83afff2700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83a80fd2e0 0x7f83a80fd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:11.609 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.600+0000 7f83afff2700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a80fdbf0 con 0x7f83a80fd2e0 2021-11-20T05:29:11.609 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.600+0000 7f83add8e700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83a80fd2e0 0x7f83a80fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:11.609 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.600+0000 7f83add8e700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83a80fd2e0 0x7f83a80fd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39544/0 (socket says 172.21.15.58:39544) 2021-11-20T05:29:11.609 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.600+0000 7f83add8e700 1 -- 172.21.15.58:0/1847471838 learned_addr learned my addr 172.21.15.58:0/1847471838 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:11.610 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.600+0000 7f83add8e700 1 -- 172.21.15.58:0/1847471838 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a80fdd30 con 0x7f83a80fd2e0 2021-11-20T05:29:11.610 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.601+0000 7f83add8e700 1 --2- 172.21.15.58:0/1847471838 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83a80fd2e0 0x7f83a80fd700 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f83a400b5f0 tx=0x7f83a4006ee0).ready entity=mon.0 client_cookie=eee7f561c1a199e0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:11.610 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.601+0000 7f83acd8c700 1 -- 172.21.15.58:0/1847471838 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a400d600 con 0x7f83a80fd2e0 2021-11-20T05:29:11.610 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.601+0000 7f83acd8c700 1 -- 172.21.15.58:0/1847471838 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f83a400dbd0 con 0x7f83a80fd2e0 2021-11-20T05:29:11.611 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.601+0000 7f83acd8c700 1 -- 172.21.15.58:0/1847471838 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a4011c00 con 0x7f83a80fd2e0 2021-11-20T05:29:11.611 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.601+0000 7f83afff2700 1 -- 172.21.15.58:0/1847471838 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83a80fd2e0 msgr2=0x7f83a80fd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:11.611 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.601+0000 7f83afff2700 1 --2- 172.21.15.58:0/1847471838 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83a80fd2e0 0x7f83a80fd700 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f83a400b5f0 tx=0x7f83a4006ee0).stop 2021-11-20T05:29:11.611 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.602+0000 7f83afff2700 1 -- 172.21.15.58:0/1847471838 shutdown_connections 2021-11-20T05:29:11.612 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.602+0000 7f83afff2700 1 --2- 172.21.15.58:0/1847471838 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83a80fd2e0 0x7f83a80fd700 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:11.612 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.602+0000 7f83afff2700 1 -- 172.21.15.58:0/1847471838 >> 172.21.15.58:0/1847471838 conn(0x7f83a80f5a80 msgr2=0x7f83a80f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:11.612 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.602+0000 7f83afff2700 1 -- 172.21.15.58:0/1847471838 shutdown_connections 2021-11-20T05:29:11.612 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.602+0000 7f83afff2700 1 -- 172.21.15.58:0/1847471838 wait complete. 2021-11-20T05:29:11.613 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.602+0000 7f83afff2700 1 Processor -- start 2021-11-20T05:29:11.613 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.602+0000 7f83afff2700 1 -- start start 2021-11-20T05:29:11.613 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.602+0000 7f83afff2700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83a80fd2e0 0x7f83a80fcbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:11.613 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.602+0000 7f83afff2700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a80fb030 con 0x7f83a80fd2e0 2021-11-20T05:29:11.614 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.603+0000 7f83add8e700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83a80fd2e0 0x7f83a80fcbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:11.614 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.603+0000 7f83add8e700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83a80fd2e0 0x7f83a80fcbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39546/0 (socket says 172.21.15.58:39546) 2021-11-20T05:29:11.614 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.603+0000 7f83add8e700 1 -- 172.21.15.58:0/1808582832 learned_addr learned my addr 172.21.15.58:0/1808582832 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:11.614 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.603+0000 7f83add8e700 1 -- 172.21.15.58:0/1808582832 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a400b200 con 0x7f83a80fd2e0 2021-11-20T05:29:11.615 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.603+0000 7f83add8e700 1 --2- 172.21.15.58:0/1808582832 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83a80fd2e0 0x7f83a80fcbb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f83a400e040 tx=0x7f83a4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:11.615 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.603+0000 7f839a7fc700 1 -- 172.21.15.58:0/1808582832 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a401b410 con 0x7f83a80fd2e0 2021-11-20T05:29:11.615 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.603+0000 7f839a7fc700 1 -- 172.21.15.58:0/1808582832 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f83a401b9e0 con 0x7f83a80fd2e0 2021-11-20T05:29:11.615 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.603+0000 7f83afff2700 1 -- 172.21.15.58:0/1808582832 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a80fb1d0 con 0x7f83a80fd2e0 2021-11-20T05:29:11.616 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.603+0000 7f839a7fc700 1 -- 172.21.15.58:0/1808582832 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a401a770 con 0x7f83a80fd2e0 2021-11-20T05:29:11.616 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.603+0000 7f83afff2700 1 -- 172.21.15.58:0/1808582832 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a80fb5e0 con 0x7f83a80fd2e0 2021-11-20T05:29:11.616 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.604+0000 7f839a7fc700 1 -- 172.21.15.58:0/1808582832 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f83a400f070 con 0x7f83a80fd2e0 2021-11-20T05:29:11.616 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.604+0000 7f839a7fc700 1 --2- 172.21.15.58:0/1808582832 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f839002fd00 0x7f83900321c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:11.616 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.604+0000 7f839a7fc700 1 -- 172.21.15.58:0/1808582832 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f83a4040ff0 con 0x7f83a80fd2e0 2021-11-20T05:29:11.617 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.604+0000 7f83afff2700 1 -- 172.21.15.58:0/1808582832 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a8059100 con 0x7f83a80fd2e0 2021-11-20T05:29:11.617 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.604+0000 7f83ad58d700 1 --2- 172.21.15.58:0/1808582832 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f839002fd00 0x7f83900321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:11.617 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.605+0000 7f83ad58d700 1 --2- 172.21.15.58:0/1808582832 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f839002fd00 0x7f83900321c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f839c0096a0 tx=0x7f839c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:11.618 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:11.609+0000 7f839a7fc700 1 -- 172.21.15.58:0/1808582832 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f83a4016070 con 0x7f83a80fd2e0 2021-11-20T05:29:12.012 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:12.012+0000 7f83afff2700 1 -- 172.21.15.58:0/1808582832 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f83a8045ad0 con 0x7f83a80fd2e0 2021-11-20T05:29:12.012 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:12.012+0000 7f839a7fc700 1 -- 172.21.15.58:0/1808582832 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f83a4024410 con 0x7f83a80fd2e0 2021-11-20T05:29:12.013 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:12.013 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: { 2021-11-20T05:29:12.013 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "fsid": "b33f8c4a-49c2-11ec-8c2c-001a4aab830c", 2021-11-20T05:29:12.016 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "health": { 2021-11-20T05:29:12.016 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-11-20T05:29:12.016 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "checks": {}, 2021-11-20T05:29:12.017 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "mutes": [] 2021-11-20T05:29:12.017 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:12.017 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-11-20T05:29:12.017 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "quorum": [ 2021-11-20T05:29:12.018 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 0 2021-11-20T05:29:12.018 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: ], 2021-11-20T05:29:12.018 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "quorum_names": [ 2021-11-20T05:29:12.018 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "smithi058" 2021-11-20T05:29:12.019 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: ], 2021-11-20T05:29:12.019 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "quorum_age": 12, 2021-11-20T05:29:12.019 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "monmap": { 2021-11-20T05:29:12.019 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:29:12.019 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-11-20T05:29:12.020 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_mons": 1 2021-11-20T05:29:12.020 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:12.020 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "osdmap": { 2021-11-20T05:29:12.020 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:29:12.021 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_osds": 0, 2021-11-20T05:29:12.025 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-11-20T05:29:12.026 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-11-20T05:29:12.026 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-11-20T05:29:12.026 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-11-20T05:29:12.026 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-11-20T05:29:12.027 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:12.027 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "pgmap": { 2021-11-20T05:29:12.027 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-11-20T05:29:12.027 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-11-20T05:29:12.028 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_pools": 0, 2021-11-20T05:29:12.028 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_objects": 0, 2021-11-20T05:29:12.028 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-11-20T05:29:12.028 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-11-20T05:29:12.028 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-11-20T05:29:12.029 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "bytes_total": 0 2021-11-20T05:29:12.029 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:12.029 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "fsmap": { 2021-11-20T05:29:12.029 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:29:12.030 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "by_rank": [], 2021-11-20T05:29:12.030 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "up:standby": 0 2021-11-20T05:29:12.030 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:12.030 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "mgrmap": { 2021-11-20T05:29:12.030 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "available": true, 2021-11-20T05:29:12.031 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-11-20T05:29:12.031 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "modules": [ 2021-11-20T05:29:12.031 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "iostat", 2021-11-20T05:29:12.031 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "restful" 2021-11-20T05:29:12.032 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: ], 2021-11-20T05:29:12.032 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "services": {} 2021-11-20T05:29:12.032 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:12.032 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "servicemap": { 2021-11-20T05:29:12.033 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:29:12.033 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "modified": "2021-11-20T05:28:54.787821+0000", 2021-11-20T05:29:12.033 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "services": {} 2021-11-20T05:29:12.033 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: }, 2021-11-20T05:29:12.034 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "progress_events": {} 2021-11-20T05:29:12.034 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: } 2021-11-20T05:29:12.034 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:12.014+0000 7f83afff2700 1 -- 172.21.15.58:0/1808582832 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f839002fd00 msgr2=0x7f83900321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:12.034 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:12.014+0000 7f83afff2700 1 --2- 172.21.15.58:0/1808582832 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f839002fd00 0x7f83900321c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f839c0096a0 tx=0x7f839c006b40).stop 2021-11-20T05:29:12.035 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:12.014+0000 7f83afff2700 1 -- 172.21.15.58:0/1808582832 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83a80fd2e0 msgr2=0x7f83a80fcbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:12.035 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:12.014+0000 7f83afff2700 1 --2- 172.21.15.58:0/1808582832 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83a80fd2e0 0x7f83a80fcbb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f83a400e040 tx=0x7f83a4006ee0).stop 2021-11-20T05:29:12.035 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:12.014+0000 7f83afff2700 1 -- 172.21.15.58:0/1808582832 shutdown_connections 2021-11-20T05:29:12.035 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:12.014+0000 7f83afff2700 1 --2- 172.21.15.58:0/1808582832 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f839002fd00 0x7f83900321c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:12.036 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:12.014+0000 7f83afff2700 1 --2- 172.21.15.58:0/1808582832 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83a80fd2e0 0x7f83a80fcbb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:12.036 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:12.014+0000 7f83afff2700 1 -- 172.21.15.58:0/1808582832 >> 172.21.15.58:0/1808582832 conn(0x7f83a80f5a80 msgr2=0x7f83a80ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:12.036 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:12.014+0000 7f83afff2700 1 -- 172.21.15.58:0/1808582832 shutdown_connections 2021-11-20T05:29:12.036 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:12.014+0000 7f83afff2700 1 -- 172.21.15.58:0/1808582832 wait complete. 2021-11-20T05:29:12.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:12 smithi058 conmon[33537]: audit 2021-11-20T05:29:12. 2021-11-20T05:29:12.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:12 smithi058 conmon[33537]: 013261+0000 mon.smithi058 (mon.0) 25 : audit [DBG] from='client.? 172.21.15.58:0/1808582832' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-11-20T05:29:14.152 INFO:teuthology.orchestra.run.smithi058.stderr:mgr is available 2021-11-20T05:29:14.152 INFO:teuthology.orchestra.run.smithi058.stderr:Enabling cephadm module... 2021-11-20T05:29:14.152 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2021-11-20T05:29:15.977 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.834+0000 7f5e64de7700 1 Processor -- start 2021-11-20T05:29:15.978 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.834+0000 7f5e64de7700 1 -- start start 2021-11-20T05:29:15.978 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.835+0000 7f5e64de7700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5e600fc090 0x7f5e600fc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:15.978 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.835+0000 7f5e64de7700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e600fc9a0 con 0x7f5e600fc090 2021-11-20T05:29:15.978 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.835+0000 7f5e5e59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5e600fc090 0x7f5e600fc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:15.979 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.835+0000 7f5e5e59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5e600fc090 0x7f5e600fc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39550/0 (socket says 172.21.15.58:39550) 2021-11-20T05:29:15.979 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.835+0000 7f5e5e59c700 1 -- 172.21.15.58:0/1610137242 learned_addr learned my addr 172.21.15.58:0/1610137242 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:15.979 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.835+0000 7f5e5e59c700 1 -- 172.21.15.58:0/1610137242 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e600fcae0 con 0x7f5e600fc090 2021-11-20T05:29:15.979 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.835+0000 7f5e5e59c700 1 --2- 172.21.15.58:0/1610137242 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5e600fc090 0x7f5e600fc4b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f5e5000b870 tx=0x7f5e50006f70).ready entity=mon.0 client_cookie=201383a1c2005562 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:15.980 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.836+0000 7f5e5d59a700 1 -- 172.21.15.58:0/1610137242 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e5000da20 con 0x7f5e600fc090 2021-11-20T05:29:15.980 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.836+0000 7f5e5d59a700 1 -- 172.21.15.58:0/1610137242 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5e50010470 con 0x7f5e600fc090 2021-11-20T05:29:15.980 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.836+0000 7f5e5d59a700 1 -- 172.21.15.58:0/1610137242 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e5000f5b0 con 0x7f5e600fc090 2021-11-20T05:29:15.980 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.836+0000 7f5e64de7700 1 -- 172.21.15.58:0/1610137242 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5e600fc090 msgr2=0x7f5e600fc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:15.981 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.836+0000 7f5e64de7700 1 --2- 172.21.15.58:0/1610137242 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5e600fc090 0x7f5e600fc4b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f5e5000b870 tx=0x7f5e50006f70).stop 2021-11-20T05:29:15.981 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.836+0000 7f5e64de7700 1 -- 172.21.15.58:0/1610137242 shutdown_connections 2021-11-20T05:29:15.981 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.836+0000 7f5e64de7700 1 --2- 172.21.15.58:0/1610137242 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5e600fc090 0x7f5e600fc4b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:15.981 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.836+0000 7f5e64de7700 1 -- 172.21.15.58:0/1610137242 >> 172.21.15.58:0/1610137242 conn(0x7f5e600f5a80 msgr2=0x7f5e600f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:15.981 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.836+0000 7f5e64de7700 1 -- 172.21.15.58:0/1610137242 shutdown_connections 2021-11-20T05:29:15.982 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.836+0000 7f5e64de7700 1 -- 172.21.15.58:0/1610137242 wait complete. 2021-11-20T05:29:15.982 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.837+0000 7f5e64de7700 1 Processor -- start 2021-11-20T05:29:15.982 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.837+0000 7f5e64de7700 1 -- start start 2021-11-20T05:29:15.982 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.837+0000 7f5e64de7700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5e600fc090 0x7f5e60108750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:15.983 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.837+0000 7f5e64de7700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e60108c40 con 0x7f5e600fc090 2021-11-20T05:29:15.983 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.837+0000 7f5e5e59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5e600fc090 0x7f5e60108750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:15.983 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.837+0000 7f5e5e59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5e600fc090 0x7f5e60108750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39552/0 (socket says 172.21.15.58:39552) 2021-11-20T05:29:15.983 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.837+0000 7f5e5e59c700 1 -- 172.21.15.58:0/2734238155 learned_addr learned my addr 172.21.15.58:0/2734238155 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:15.984 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.837+0000 7f5e5e59c700 1 -- 172.21.15.58:0/2734238155 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5000b480 con 0x7f5e600fc090 2021-11-20T05:29:15.984 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.837+0000 7f5e5e59c700 1 --2- 172.21.15.58:0/2734238155 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5e600fc090 0x7f5e60108750 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f5e50006b80 tx=0x7f5e50006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:15.984 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.838+0000 7f5e4effd700 1 -- 172.21.15.58:0/2734238155 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e5000f3d0 con 0x7f5e600fc090 2021-11-20T05:29:15.985 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.838+0000 7f5e4effd700 1 -- 172.21.15.58:0/2734238155 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5e5000fef0 con 0x7f5e600fc090 2021-11-20T05:29:15.985 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.838+0000 7f5e64de7700 1 -- 172.21.15.58:0/2734238155 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e60108e40 con 0x7f5e600fc090 2021-11-20T05:29:15.985 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.838+0000 7f5e4effd700 1 -- 172.21.15.58:0/2734238155 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e50019c70 con 0x7f5e600fc090 2021-11-20T05:29:15.985 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.838+0000 7f5e64de7700 1 -- 172.21.15.58:0/2734238155 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e60109250 con 0x7f5e600fc090 2021-11-20T05:29:15.985 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.839+0000 7f5e4effd700 1 -- 172.21.15.58:0/2734238155 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f5e50016030 con 0x7f5e600fc090 2021-11-20T05:29:15.986 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.839+0000 7f5e64de7700 1 -- 172.21.15.58:0/2734238155 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e60045ad0 con 0x7f5e600fc090 2021-11-20T05:29:15.986 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.839+0000 7f5e4effd700 1 --2- 172.21.15.58:0/2734238155 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f5e4402fd50 0x7f5e44032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:15.986 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.839+0000 7f5e4effd700 1 -- 172.21.15.58:0/2734238155 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5e50044090 con 0x7f5e600fc090 2021-11-20T05:29:15.987 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.839+0000 7f5e5dd9b700 1 --2- 172.21.15.58:0/2734238155 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f5e4402fd50 0x7f5e44032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:15.987 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.840+0000 7f5e5dd9b700 1 --2- 172.21.15.58:0/2734238155 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f5e4402fd50 0x7f5e44032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5e540096a0 tx=0x7f5e54006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:15.987 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:15.843+0000 7f5e4effd700 1 -- 172.21.15.58:0/2734238155 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e50022370 con 0x7f5e600fc090 2021-11-20T05:29:16.200 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:16.200+0000 7f5e64de7700 1 -- 172.21.15.58:0/2734238155 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f5e600fa6e0 con 0x7f5e600fc090 2021-11-20T05:29:16.252 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:16.252+0000 7f5e4effd700 1 -- 172.21.15.58:0/2734238155 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f5e500233c0 con 0x7f5e600fc090 2021-11-20T05:29:16.253 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:16.254+0000 7f5e4effd700 1 -- 172.21.15.58:0/2734238155 <== mon.0 v2:172.21.15.58:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f5e50022370 con 0x7f5e600fc090 2021-11-20T05:29:16.258 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:16.256+0000 7f5e64de7700 1 -- 172.21.15.58:0/2734238155 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f5e4402fd50 msgr2=0x7f5e44032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:16.259 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:16.256+0000 7f5e64de7700 1 --2- 172.21.15.58:0/2734238155 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f5e4402fd50 0x7f5e44032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5e540096a0 tx=0x7f5e54006b40).stop 2021-11-20T05:29:16.259 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:16.256+0000 7f5e64de7700 1 -- 172.21.15.58:0/2734238155 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5e600fc090 msgr2=0x7f5e60108750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:16.259 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:16.256+0000 7f5e64de7700 1 --2- 172.21.15.58:0/2734238155 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5e600fc090 0x7f5e60108750 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f5e50006b80 tx=0x7f5e50006f70).stop 2021-11-20T05:29:16.259 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:16.257+0000 7f5e64de7700 1 -- 172.21.15.58:0/2734238155 shutdown_connections 2021-11-20T05:29:16.260 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:16.257+0000 7f5e64de7700 1 --2- 172.21.15.58:0/2734238155 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f5e4402fd50 0x7f5e44032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:16.260 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:16.257+0000 7f5e64de7700 1 --2- 172.21.15.58:0/2734238155 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5e600fc090 0x7f5e60108750 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:16.260 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:16.257+0000 7f5e64de7700 1 -- 172.21.15.58:0/2734238155 >> 172.21.15.58:0/2734238155 conn(0x7f5e600f5a80 msgr2=0x7f5e600ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:16.260 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:16.257+0000 7f5e64de7700 1 -- 172.21.15.58:0/2734238155 shutdown_connections 2021-11-20T05:29:16.261 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:16.257+0000 7f5e64de7700 1 -- 172.21.15.58:0/2734238155 wait complete. 2021-11-20T05:29:16.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:16 smithi058 conmon[33537]: audit 2021-11-20T05:29:16.201737+0000 mon.smithi058 (mon.0) 26 : audit [INF] from='client.? 172.21.15.58:0/2734238155' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2021-11-20T05:29:17.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:17 smithi058 conmon[33537]: audit 2021-11-20T05:29:16.254929+0000 mon.smithi058 (mon.0) 27 : audit [INF] from='client.? 172.21.15.58:0/2734238155' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2021-11-20T05:29:17.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:17 smithi058 conmon[33537]: cluster 2021-11-20T05:29:16.254993+0000 mon.smithi058 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi058.zagdlk(active, since 9s) 2021-11-20T05:29:18.302 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2021-11-20T05:29:20.321 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.293+0000 7f9d69d30700 1 Processor -- start 2021-11-20T05:29:20.321 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.293+0000 7f9d69d30700 1 -- start start 2021-11-20T05:29:20.321 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.293+0000 7f9d69d30700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9d6406fc50 0x7f9d640fe820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:20.322 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.293+0000 7f9d69d30700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d640feda0 con 0x7f9d6406fc50 2021-11-20T05:29:20.322 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.294+0000 7f9d637fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9d6406fc50 0x7f9d640fe820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:20.322 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.294+0000 7f9d637fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9d6406fc50 0x7f9d640fe820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39570/0 (socket says 172.21.15.58:39570) 2021-11-20T05:29:20.322 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.294+0000 7f9d637fe700 1 -- 172.21.15.58:0/3268375074 learned_addr learned my addr 172.21.15.58:0/3268375074 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:20.323 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.294+0000 7f9d637fe700 1 -- 172.21.15.58:0/3268375074 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d640feee0 con 0x7f9d6406fc50 2021-11-20T05:29:20.323 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.294+0000 7f9d637fe700 1 --2- 172.21.15.58:0/3268375074 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9d6406fc50 0x7f9d640fe820 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9d4c00b5f0 tx=0x7f9d4c006ee0).ready entity=mon.0 client_cookie=9443fae8250da2b0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:20.323 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.294+0000 7f9d627fc700 1 -- 172.21.15.58:0/3268375074 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d4c00d600 con 0x7f9d6406fc50 2021-11-20T05:29:20.323 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.294+0000 7f9d627fc700 1 -- 172.21.15.58:0/3268375074 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9d4c00dbd0 con 0x7f9d6406fc50 2021-11-20T05:29:20.324 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.294+0000 7f9d627fc700 1 -- 172.21.15.58:0/3268375074 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d4c011c00 con 0x7f9d6406fc50 2021-11-20T05:29:20.324 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.295+0000 7f9d69d30700 1 -- 172.21.15.58:0/3268375074 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9d6406fc50 msgr2=0x7f9d640fe820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:20.324 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.295+0000 7f9d69d30700 1 --2- 172.21.15.58:0/3268375074 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9d6406fc50 0x7f9d640fe820 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9d4c00b5f0 tx=0x7f9d4c006ee0).stop 2021-11-20T05:29:20.324 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.295+0000 7f9d69d30700 1 -- 172.21.15.58:0/3268375074 shutdown_connections 2021-11-20T05:29:20.324 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.295+0000 7f9d69d30700 1 --2- 172.21.15.58:0/3268375074 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9d6406fc50 0x7f9d640fe820 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:20.325 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.295+0000 7f9d69d30700 1 -- 172.21.15.58:0/3268375074 >> 172.21.15.58:0/3268375074 conn(0x7f9d640f6280 msgr2=0x7f9d640f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:20.325 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.295+0000 7f9d69d30700 1 -- 172.21.15.58:0/3268375074 shutdown_connections 2021-11-20T05:29:20.325 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.295+0000 7f9d69d30700 1 -- 172.21.15.58:0/3268375074 wait complete. 2021-11-20T05:29:20.325 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.296+0000 7f9d69d30700 1 Processor -- start 2021-11-20T05:29:20.326 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.296+0000 7f9d69d30700 1 -- start start 2021-11-20T05:29:20.326 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.296+0000 7f9d69d30700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9d6406fc50 0x7f9d6410a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:20.326 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.296+0000 7f9d69d30700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6410aeb0 con 0x7f9d6406fc50 2021-11-20T05:29:20.327 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.296+0000 7f9d637fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9d6406fc50 0x7f9d6410a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:20.327 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.296+0000 7f9d637fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9d6406fc50 0x7f9d6410a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39572/0 (socket says 172.21.15.58:39572) 2021-11-20T05:29:20.328 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.296+0000 7f9d637fe700 1 -- 172.21.15.58:0/1764295854 learned_addr learned my addr 172.21.15.58:0/1764295854 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:20.328 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.297+0000 7f9d637fe700 1 -- 172.21.15.58:0/1764295854 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c00b200 con 0x7f9d6406fc50 2021-11-20T05:29:20.328 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.297+0000 7f9d637fe700 1 --2- 172.21.15.58:0/1764295854 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9d6406fc50 0x7f9d6410a9c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f9d4c000c00 tx=0x7f9d4c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:20.328 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.297+0000 7f9d5bfff700 1 -- 172.21.15.58:0/1764295854 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d4c011690 con 0x7f9d6406fc50 2021-11-20T05:29:20.329 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.297+0000 7f9d5bfff700 1 -- 172.21.15.58:0/1764295854 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9d4c00d9f0 con 0x7f9d6406fc50 2021-11-20T05:29:20.329 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.297+0000 7f9d69d30700 1 -- 172.21.15.58:0/1764295854 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d6410b0b0 con 0x7f9d6406fc50 2021-11-20T05:29:20.329 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.297+0000 7f9d5bfff700 1 -- 172.21.15.58:0/1764295854 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d4c01b710 con 0x7f9d6406fc50 2021-11-20T05:29:20.329 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.297+0000 7f9d69d30700 1 -- 172.21.15.58:0/1764295854 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6410b4c0 con 0x7f9d6406fc50 2021-11-20T05:29:20.330 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.298+0000 7f9d5bfff700 1 -- 172.21.15.58:0/1764295854 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9d4c00f070 con 0x7f9d6406fc50 2021-11-20T05:29:20.330 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.298+0000 7f9d5bfff700 1 --2- 172.21.15.58:0/1764295854 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f9d4402fe00 0x7f9d440322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:20.330 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.298+0000 7f9d69d30700 1 -- 172.21.15.58:0/1764295854 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d64045ad0 con 0x7f9d6406fc50 2021-11-20T05:29:20.330 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.298+0000 7f9d5bfff700 1 -- 172.21.15.58:0/1764295854 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9d4c041130 con 0x7f9d6406fc50 2021-11-20T05:29:20.330 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.298+0000 7f9d62ffd700 1 -- 172.21.15.58:0/1764295854 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f9d4402fe00 msgr2=0x7f9d440322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.58:6800/324085383 2021-11-20T05:29:20.331 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.298+0000 7f9d62ffd700 1 --2- 172.21.15.58:0/1764295854 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f9d4402fe00 0x7f9d440322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T05:29:20.331 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.303+0000 7f9d5bfff700 1 -- 172.21.15.58:0/1764295854 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d4c0224f0 con 0x7f9d6406fc50 2021-11-20T05:29:20.499 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.499+0000 7f9d62ffd700 1 -- 172.21.15.58:0/1764295854 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f9d4402fe00 msgr2=0x7f9d440322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.58:6800/324085383 2021-11-20T05:29:20.499 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.499+0000 7f9d62ffd700 1 --2- 172.21.15.58:0/1764295854 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f9d4402fe00 0x7f9d440322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T05:29:20.664 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.662+0000 7f9d69d30700 1 -- 172.21.15.58:0/1764295854 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f9d6410c5d0 con 0x7f9d6406fc50 2021-11-20T05:29:20.665 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.663+0000 7f9d5bfff700 1 -- 172.21.15.58:0/1764295854 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f9d4c022310 con 0x7f9d6406fc50 2021-11-20T05:29:20.665 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: { 2021-11-20T05:29:20.666 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 5, 2021-11-20T05:29:20.666 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "available": true, 2021-11-20T05:29:20.666 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "active_name": "smithi058.zagdlk", 2021-11-20T05:29:20.667 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_standby": 0 2021-11-20T05:29:20.667 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: } 2021-11-20T05:29:20.670 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.665+0000 7f9d69d30700 1 -- 172.21.15.58:0/1764295854 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f9d4402fe00 msgr2=0x7f9d440322c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T05:29:20.670 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.665+0000 7f9d69d30700 1 --2- 172.21.15.58:0/1764295854 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f9d4402fe00 0x7f9d440322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:29:20.671 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.665+0000 7f9d69d30700 1 -- 172.21.15.58:0/1764295854 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9d6406fc50 msgr2=0x7f9d6410a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:20.671 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.665+0000 7f9d69d30700 1 --2- 172.21.15.58:0/1764295854 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9d6406fc50 0x7f9d6410a9c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f9d4c000c00 tx=0x7f9d4c006ee0).stop 2021-11-20T05:29:20.671 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.665+0000 7f9d69d30700 1 -- 172.21.15.58:0/1764295854 shutdown_connections 2021-11-20T05:29:20.672 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.665+0000 7f9d69d30700 1 --2- 172.21.15.58:0/1764295854 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7f9d4402fe00 0x7f9d440322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:29:20.672 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.665+0000 7f9d69d30700 1 --2- 172.21.15.58:0/1764295854 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9d6406fc50 0x7f9d6410a9c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:20.673 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.665+0000 7f9d69d30700 1 -- 172.21.15.58:0/1764295854 >> 172.21.15.58:0/1764295854 conn(0x7f9d640f6280 msgr2=0x7f9d640f7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:20.673 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.666+0000 7f9d69d30700 1 -- 172.21.15.58:0/1764295854 shutdown_connections 2021-11-20T05:29:20.674 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:20.666+0000 7f9d69d30700 1 -- 172.21.15.58:0/1764295854 wait complete. 2021-11-20T05:29:20.969 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:20 smithi058 conmon[33537]: audit 2021-11-20T05:29:20 2021-11-20T05:29:20.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:20 smithi058 conmon[33537]: .663973+0000 mon.smithi058 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.58:0/1764295854' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-11-20T05:29:21.043 INFO:teuthology.orchestra.run.smithi058.stderr:Waiting for the mgr to restart... 2021-11-20T05:29:21.043 INFO:teuthology.orchestra.run.smithi058.stderr:Waiting for mgr epoch 5... 2021-11-20T05:29:21.044 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2021-11-20T05:29:21.520 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.518+0000 7fb53d43a700 1 Processor -- start 2021-11-20T05:29:21.521 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.518+0000 7fb53d43a700 1 -- start start 2021-11-20T05:29:21.521 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.518+0000 7fb53d43a700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5380fbbc0 0x7fb5380fbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:21.521 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.518+0000 7fb53d43a700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380fc4d0 con 0x7fb5380fbbc0 2021-11-20T05:29:21.521 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.518+0000 7fb536ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5380fbbc0 0x7fb5380fbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:21.522 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.518+0000 7fb536ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5380fbbc0 0x7fb5380fbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39578/0 (socket says 172.21.15.58:39578) 2021-11-20T05:29:21.522 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.518+0000 7fb536ffd700 1 -- 172.21.15.58:0/2950313697 learned_addr learned my addr 172.21.15.58:0/2950313697 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:21.522 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.519+0000 7fb536ffd700 1 -- 172.21.15.58:0/2950313697 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5380fc610 con 0x7fb5380fbbc0 2021-11-20T05:29:21.522 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.519+0000 7fb536ffd700 1 --2- 172.21.15.58:0/2950313697 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5380fbbc0 0x7fb5380fbfe0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fb52800b5f0 tx=0x7fb528006ee0).ready entity=mon.0 client_cookie=f05d5425bd0d4c07 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:21.523 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.519+0000 7fb535ffb700 1 -- 172.21.15.58:0/2950313697 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb52800d600 con 0x7fb5380fbbc0 2021-11-20T05:29:21.523 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.519+0000 7fb535ffb700 1 -- 172.21.15.58:0/2950313697 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb52800dbd0 con 0x7fb5380fbbc0 2021-11-20T05:29:21.523 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.519+0000 7fb535ffb700 1 -- 172.21.15.58:0/2950313697 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb528011c00 con 0x7fb5380fbbc0 2021-11-20T05:29:21.523 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.520+0000 7fb53d43a700 1 -- 172.21.15.58:0/2950313697 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5380fbbc0 msgr2=0x7fb5380fbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:21.524 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.520+0000 7fb53d43a700 1 --2- 172.21.15.58:0/2950313697 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5380fbbc0 0x7fb5380fbfe0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fb52800b5f0 tx=0x7fb528006ee0).stop 2021-11-20T05:29:21.524 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.520+0000 7fb53d43a700 1 -- 172.21.15.58:0/2950313697 shutdown_connections 2021-11-20T05:29:21.524 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.520+0000 7fb53d43a700 1 --2- 172.21.15.58:0/2950313697 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5380fbbc0 0x7fb5380fbfe0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:21.524 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.520+0000 7fb53d43a700 1 -- 172.21.15.58:0/2950313697 >> 172.21.15.58:0/2950313697 conn(0x7fb5380f73e0 msgr2=0x7fb5380f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:21.524 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.520+0000 7fb53d43a700 1 -- 172.21.15.58:0/2950313697 shutdown_connections 2021-11-20T05:29:21.525 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.520+0000 7fb53d43a700 1 -- 172.21.15.58:0/2950313697 wait complete. 2021-11-20T05:29:21.525 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.521+0000 7fb53d43a700 1 Processor -- start 2021-11-20T05:29:21.525 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.521+0000 7fb53d43a700 1 -- start start 2021-11-20T05:29:21.528 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.521+0000 7fb53d43a700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5380fbbc0 0x7fb538071d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:21.528 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.521+0000 7fb53d43a700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380701f0 con 0x7fb5380fbbc0 2021-11-20T05:29:21.528 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.521+0000 7fb536ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5380fbbc0 0x7fb538071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:21.528 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.522+0000 7fb536ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5380fbbc0 0x7fb538071d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39580/0 (socket says 172.21.15.58:39580) 2021-11-20T05:29:21.529 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.522+0000 7fb536ffd700 1 -- 172.21.15.58:0/2334087079 learned_addr learned my addr 172.21.15.58:0/2334087079 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:21.529 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.522+0000 7fb536ffd700 1 -- 172.21.15.58:0/2334087079 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52800b200 con 0x7fb5380fbbc0 2021-11-20T05:29:21.529 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.522+0000 7fb536ffd700 1 --2- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5380fbbc0 0x7fb538071d70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fb52800b1d0 tx=0x7fb528006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:21.529 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.522+0000 7fb51f7fe700 1 -- 172.21.15.58:0/2334087079 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb52801b410 con 0x7fb5380fbbc0 2021-11-20T05:29:21.529 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.522+0000 7fb51f7fe700 1 -- 172.21.15.58:0/2334087079 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb52801b9e0 con 0x7fb5380fbbc0 2021-11-20T05:29:21.530 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.522+0000 7fb51f7fe700 1 -- 172.21.15.58:0/2334087079 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb52801a840 con 0x7fb5380fbbc0 2021-11-20T05:29:21.530 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.523+0000 7fb53d43a700 1 -- 172.21.15.58:0/2334087079 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb538070390 con 0x7fb5380fbbc0 2021-11-20T05:29:21.530 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.523+0000 7fb53d43a700 1 -- 172.21.15.58:0/2334087079 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5380707a0 con 0x7fb5380fbbc0 2021-11-20T05:29:21.530 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.524+0000 7fb51f7fe700 1 -- 172.21.15.58:0/2334087079 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb52800f070 con 0x7fb5380fbbc0 2021-11-20T05:29:21.531 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.524+0000 7fb51f7fe700 1 --2- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7fb52402fd50 0x7fb524032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:21.531 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.524+0000 7fb51f7fe700 1 -- 172.21.15.58:0/2334087079 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb5280407f0 con 0x7fb5380fbbc0 2021-11-20T05:29:21.531 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.524+0000 7fb5367fc700 1 -- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7fb52402fd50 msgr2=0x7fb524032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.58:6800/324085383 2021-11-20T05:29:21.531 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.524+0000 7fb5367fc700 1 --2- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7fb52402fd50 0x7fb524032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T05:29:21.532 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.524+0000 7fb53d43a700 1 -- 172.21.15.58:0/2334087079 --> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb538059100 con 0x7fb52402fd50 2021-11-20T05:29:21.725 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.725+0000 7fb5367fc700 1 -- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7fb52402fd50 msgr2=0x7fb524032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.58:6800/324085383 2021-11-20T05:29:21.725 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:21.725+0000 7fb5367fc700 1 --2- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7fb52402fd50 0x7fb524032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T05:29:22.126 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:22.126+0000 7fb5367fc700 1 -- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7fb52402fd50 msgr2=0x7fb524032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.58:6800/324085383 2021-11-20T05:29:22.126 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:22.126+0000 7fb5367fc700 1 --2- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7fb52402fd50 0x7fb524032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-20T05:29:22.318 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:22.318+0000 7fb51f7fe700 1 -- 172.21.15.58:0/2334087079 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7fb52801bb50 con 0x7fb5380fbbc0 2021-11-20T05:29:22.318 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:22.318+0000 7fb51f7fe700 1 -- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7fb52402fd50 msgr2=0x7fb524032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T05:29:22.318 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:22.318+0000 7fb51f7fe700 1 --2- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7fb52402fd50 0x7fb524032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:29:23.320 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.319+0000 7fb51f7fe700 1 -- 172.21.15.58:0/2334087079 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fb528010360 con 0x7fb5380fbbc0 2021-11-20T05:29:23.320 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.319+0000 7fb51f7fe700 1 --2- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fb524033670 0x7fb524035a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:23.320 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.319+0000 7fb51f7fe700 1 -- 172.21.15.58:0/2334087079 --> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb538059100 con 0x7fb524033670 2021-11-20T05:29:23.321 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.320+0000 7fb5367fc700 1 --2- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fb524033670 0x7fb524035a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:23.321 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.321+0000 7fb5367fc700 1 --2- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fb524033670 0x7fb524035a60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fb52c0036d0 tx=0x7fb52c006c50).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:23.322 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.322+0000 7fb51f7fe700 1 -- 172.21.15.58:0/2334087079 <== mgr.14116 v2:172.21.15.58:6800/1261811851 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fb538059100 con 0x7fb524033670 2021-11-20T05:29:23.336 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.332+0000 7fb53d43a700 1 -- 172.21.15.58:0/2334087079 --> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb538045ad0 con 0x7fb524033670 2021-11-20T05:29:23.336 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.333+0000 7fb51f7fe700 1 -- 172.21.15.58:0/2334087079 <== mgr.14116 v2:172.21.15.58:6800/1261811851 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fb538045ad0 con 0x7fb524033670 2021-11-20T05:29:23.336 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: { 2021-11-20T05:29:23.337 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2021-11-20T05:29:23.337 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "initialized": true 2021-11-20T05:29:23.337 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: } 2021-11-20T05:29:23.338 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.333+0000 7fb53d43a700 1 -- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fb524033670 msgr2=0x7fb524035a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:23.338 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.333+0000 7fb53d43a700 1 --2- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fb524033670 0x7fb524035a60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fb52c0036d0 tx=0x7fb52c006c50).stop 2021-11-20T05:29:23.338 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.333+0000 7fb53d43a700 1 -- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5380fbbc0 msgr2=0x7fb538071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:23.338 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.333+0000 7fb53d43a700 1 --2- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5380fbbc0 0x7fb538071d70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fb52800b1d0 tx=0x7fb528006ee0).stop 2021-11-20T05:29:23.338 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.333+0000 7fb53d43a700 1 -- 172.21.15.58:0/2334087079 shutdown_connections 2021-11-20T05:29:23.339 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.333+0000 7fb53d43a700 1 --2- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fb524033670 0x7fb524035a60 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:23.339 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.333+0000 7fb53d43a700 1 --2- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5380fbbc0 0x7fb538071d70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:23.339 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.333+0000 7fb53d43a700 1 --2- 172.21.15.58:0/2334087079 >> [v2:172.21.15.58:6800/324085383,v1:172.21.15.58:6801/324085383] conn(0x7fb52402fd50 0x7fb524032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:29:23.339 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.333+0000 7fb53d43a700 1 -- 172.21.15.58:0/2334087079 >> 172.21.15.58:0/2334087079 conn(0x7fb5380f73e0 msgr2=0x7fb5380f8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:23.340 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.334+0000 7fb53d43a700 1 -- 172.21.15.58:0/2334087079 shutdown_connections 2021-11-20T05:29:23.340 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:23.334+0000 7fb53d43a700 1 -- 172.21.15.58:0/2334087079 wait complete. 2021-11-20T05:29:23.685 INFO:teuthology.orchestra.run.smithi058.stderr:mgr epoch 5 is available 2021-11-20T05:29:23.685 INFO:teuthology.orchestra.run.smithi058.stderr:Setting orchestrator backend to cephadm... 2021-11-20T05:29:23.686 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2021-11-20T05:29:23.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: cluster 2021-11-20T05:29:22.266806+0000 mon.smithi058 (mon.0) 30 : cluster [INF] Active manager daemon smithi058.zagdlk restarted 2021-11-20T05:29:23.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: cluster 2021-11-20T05:29:22.267211+0000 mon.smithi058 (mon.0) 31 : cluster [INF] Activating manager daemon smithi058.zagdlk 2021-11-20T05:29:23.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: cluster 2021-11-20T05:29:22.269203+0000 mon.smithi058 (mon.0) 32 : cluster 2021-11-20T05:29:23.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: [DBG] osdmap e2: 0 total, 0 up, 0 in 2021-11-20T05:29:23.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: cluster 2021-11-20T05 2021-11-20T05:29:23.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: :29:22.319917+0000 mon.smithi058 (mon.0) 33 : cluster 2021-11-20T05:29:23.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: [DBG] mgrmap e6: smithi058.zagdlk(active, starting, since 0.0527786s) 2021-11-20T05:29:23.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: audit 2021-11-20T05 2021-11-20T05:29:23.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: :29:22.320768+0000 mon.smithi058 (mon. 2021-11-20T05:29:23.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: 0) 34 : audit [DBG] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi058"}]: dispatch 2021-11-20T05:29:23.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: audit 2021-11- 2021-11-20T05:29:23.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: 20T05:29:22.321472+0000 mon.smithi058 ( 2021-11-20T05:29:23.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mgr metadata", "who": "smithi058.zagdlk", "id": "smithi058.zagdlk"}]: dispatch 2021-11-20T05:29:23.724 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:29:23.725 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: 11-20T05:29:22.322181+0000 2021-11-20T05:29:23.725 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: mon.smithi058 (mon.0) 36 : audit [DBG] 2021-11-20T05:29:23.725 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-20T05:29:23.725 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: audit 2021-11-20T05:29:23.726 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: 2021-11-20T05:29:22.322370+0000 2021-11-20T05:29:23.727 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: mon.smithi058 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-20T05:29:23.727 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: 2021-11-20T05:29:23.727 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: audit 2021-11-20T05:29:23.727 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: 2021-11-20T05:29:22.322518+0000 mon.smithi058 2021-11-20T05:29:23.728 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-20T05:29:23.728 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: 2021-11-20T05:29:23.728 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:29:23.728 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: -11-20T05:29:22.336391+0000 mon.smithi058 2021-11-20T05:29:23.729 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: (mon.0) 39 : cluster [INF] Manager daemon smithi058.zagdlk is now available 2021-11-20T05:29:23.729 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: 2021-11-20T05:29:23.729 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: audit 2021-11-20T05:29:23.730 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: 2021-11-20T05:29:22.352690 2021-11-20T05:29:23.730 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 40 : audit [INF] 2021-11-20T05:29:23.730 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:23.730 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: audit 2021-11-20T05:29: 2021-11-20T05:29:23.731 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: 22.355382+0000 mon.smithi058 (mon.0) 41 2021-11-20T05:29:23.731 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: : audit [INF] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:23.731 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: audit 2021-11-20 2021-11-20T05:29:23.731 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: T05:29:22.357919+0000 mon.smithi058 ( 2021-11-20T05:29:23.731 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:23.732 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: audit 2021 2021-11-20T05:29:23.732 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: -11-20T05:29:22.480576 2021-11-20T05:29:23.732 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 43 : audit 2021-11-20T05:29:23.732 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: [INF] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:23.732 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: audit 2021-11-20T05: 2021-11-20T05:29:23.733 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: 29:22.481269+0000 mon.smithi058 (mon.0 2021-11-20T05:29:23.733 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: ) 44 : audit [DBG] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:29:23.733 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:29:23.733 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: -20T05:29:22.483245+0000 2021-11-20T05:29:23.734 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: mon.smithi058 (mon.0) 45 : audit [DBG] 2021-11-20T05:29:23.734 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:29:23.734 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: audit 2021-11-20T05:29: 2021-11-20T05:29:23.734 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: 22.484945+0000 mon.smithi058 (mon.0) 46 2021-11-20T05:29:23.734 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: : audit [DBG] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:29:23.735 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: audit 2021-11-20T 2021-11-20T05:29:23.735 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: 05:29:22.495915+0000 mon.smithi058 (mon 2021-11-20T05:29:23.735 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: .0) 47 : audit [INF] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:29:23.735 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:29:23.736 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: 11-20T05:29:22.497250+0000 2021-11-20T05:29:23.736 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: mon.smithi058 (mon.0) 48 : audit [INF] 2021-11-20T05:29:23.736 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:23 smithi058 conmon[33537]: from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:29:24.326 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.322+0000 7fa87979b700 1 Processor -- start 2021-11-20T05:29:24.326 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.322+0000 7fa87979b700 1 -- start start 2021-11-20T05:29:24.327 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.322+0000 7fa87979b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa87406fcc0 0x7fa874100c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:24.327 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.322+0000 7fa87979b700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8740fc650 con 0x7fa87406fcc0 2021-11-20T05:29:24.327 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.323+0000 7fa872ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa87406fcc0 0x7fa874100c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:24.327 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.323+0000 7fa872ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa87406fcc0 0x7fa874100c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39610/0 (socket says 172.21.15.58:39610) 2021-11-20T05:29:24.328 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.323+0000 7fa872ffd700 1 -- 172.21.15.58:0/1801269564 learned_addr learned my addr 172.21.15.58:0/1801269564 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:24.328 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.323+0000 7fa872ffd700 1 -- 172.21.15.58:0/1801269564 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa874101150 con 0x7fa87406fcc0 2021-11-20T05:29:24.328 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.323+0000 7fa872ffd700 1 --2- 172.21.15.58:0/1801269564 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa87406fcc0 0x7fa874100c60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fa86c009960 tx=0x7fa86c0092d0).ready entity=mon.0 client_cookie=dceef489a438c101 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:24.328 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.323+0000 7fa871ffb700 1 -- 172.21.15.58:0/1801269564 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa86c00b9c0 con 0x7fa87406fcc0 2021-11-20T05:29:24.328 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.323+0000 7fa871ffb700 1 -- 172.21.15.58:0/1801269564 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa86c011400 con 0x7fa87406fcc0 2021-11-20T05:29:24.329 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.323+0000 7fa871ffb700 1 -- 172.21.15.58:0/1801269564 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa86c0103a0 con 0x7fa87406fcc0 2021-11-20T05:29:24.329 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.324+0000 7fa87979b700 1 -- 172.21.15.58:0/1801269564 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa87406fcc0 msgr2=0x7fa874100c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:24.329 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.324+0000 7fa87979b700 1 --2- 172.21.15.58:0/1801269564 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa87406fcc0 0x7fa874100c60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fa86c009960 tx=0x7fa86c0092d0).stop 2021-11-20T05:29:24.329 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.324+0000 7fa87979b700 1 -- 172.21.15.58:0/1801269564 shutdown_connections 2021-11-20T05:29:24.330 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.324+0000 7fa87979b700 1 --2- 172.21.15.58:0/1801269564 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa87406fcc0 0x7fa874100c60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:24.330 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.324+0000 7fa87979b700 1 -- 172.21.15.58:0/1801269564 >> 172.21.15.58:0/1801269564 conn(0x7fa8740f62e0 msgr2=0x7fa8740f8700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:24.330 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.324+0000 7fa87979b700 1 -- 172.21.15.58:0/1801269564 shutdown_connections 2021-11-20T05:29:24.330 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.324+0000 7fa87979b700 1 -- 172.21.15.58:0/1801269564 wait complete. 2021-11-20T05:29:24.331 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.325+0000 7fa87979b700 1 Processor -- start 2021-11-20T05:29:24.331 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.325+0000 7fa87979b700 1 -- start start 2021-11-20T05:29:24.332 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.325+0000 7fa87979b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa87406fcc0 0x7fa87410cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:24.332 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.325+0000 7fa87979b700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87410d210 con 0x7fa87406fcc0 2021-11-20T05:29:24.332 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.325+0000 7fa872ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa87406fcc0 0x7fa87410cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:24.332 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.325+0000 7fa872ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa87406fcc0 0x7fa87410cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39612/0 (socket says 172.21.15.58:39612) 2021-11-20T05:29:24.332 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.325+0000 7fa872ffd700 1 -- 172.21.15.58:0/818981686 learned_addr learned my addr 172.21.15.58:0/818981686 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:24.333 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.326+0000 7fa872ffd700 1 -- 172.21.15.58:0/818981686 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86c009570 con 0x7fa87406fcc0 2021-11-20T05:29:24.333 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.326+0000 7fa872ffd700 1 --2- 172.21.15.58:0/818981686 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa87406fcc0 0x7fa87410cd20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fa86c006dc0 tx=0x7fa86c0092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:24.333 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.326+0000 7fa86b7fe700 1 -- 172.21.15.58:0/818981686 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa86c010770 con 0x7fa87406fcc0 2021-11-20T05:29:24.333 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.326+0000 7fa87979b700 1 -- 172.21.15.58:0/818981686 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87410d410 con 0x7fa87406fcc0 2021-11-20T05:29:24.334 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.326+0000 7fa86b7fe700 1 -- 172.21.15.58:0/818981686 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa86c010d40 con 0x7fa87406fcc0 2021-11-20T05:29:24.334 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.326+0000 7fa86b7fe700 1 -- 172.21.15.58:0/818981686 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa86c01adc0 con 0x7fa87406fcc0 2021-11-20T05:29:24.334 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.326+0000 7fa87979b700 1 -- 172.21.15.58:0/818981686 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87410d820 con 0x7fa87406fcc0 2021-11-20T05:29:24.334 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.327+0000 7fa86b7fe700 1 -- 172.21.15.58:0/818981686 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fa86c00f070 con 0x7fa87406fcc0 2021-11-20T05:29:24.335 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.327+0000 7fa86b7fe700 1 --2- 172.21.15.58:0/818981686 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fa85402fcc0 0x7fa854032180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:24.336 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.327+0000 7fa86b7fe700 1 -- 172.21.15.58:0/818981686 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa86c041cd0 con 0x7fa87406fcc0 2021-11-20T05:29:24.336 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.327+0000 7fa8727fc700 1 --2- 172.21.15.58:0/818981686 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fa85402fcc0 0x7fa854032180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:24.336 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.327+0000 7fa8727fc700 1 --2- 172.21.15.58:0/818981686 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fa85402fcc0 0x7fa854032180 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa85c0096a0 tx=0x7fa85c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:24.336 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.327+0000 7fa87979b700 1 -- 172.21.15.58:0/818981686 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa874059100 con 0x7fa87406fcc0 2021-11-20T05:29:24.337 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.332+0000 7fa86b7fe700 1 -- 172.21.15.58:0/818981686 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa86c01e070 con 0x7fa87406fcc0 2021-11-20T05:29:24.675 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:24 smithi058 conmon[33537]: audit 2021-11-20T05:29:23.322487+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:29:24.675 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:24 smithi058 conmon[33537]: mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-11-20T05:29:24.676 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:24 smithi058 conmon[33537]: cluster 2021-11-20T05:29:23.322989+0000 mon.smithi058 (mon.0) 49 : cluster [DBG] 2021-11-20T05:29:24.676 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:24 smithi058 conmon[33537]: mgrmap e7: smithi058.zagdlk(active, since 1.05585s) 2021-11-20T05:29:24.676 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:24 smithi058 conmon[33537]: audit 2021-11- 2021-11-20T05:29:24.676 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:24 smithi058 conmon[33537]: 20T05:29:23.333978+0000 mgr.smithi058.zagdlk (mgr.14116 2021-11-20T05:29:24.677 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:24 smithi058 conmon[33537]: ) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-11-20T05:29:24.677 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.675+0000 7fa87979b700 1 -- 172.21.15.58:0/818981686 --> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fa87410ec70 con 0x7fa85402fcc0 2021-11-20T05:29:24.680 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.680+0000 7fa86b7fe700 1 -- 172.21.15.58:0/818981686 <== mgr.14116 v2:172.21.15.58:6800/1261811851 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa87410ec70 con 0x7fa85402fcc0 2021-11-20T05:29:24.683 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.682+0000 7fa87979b700 1 -- 172.21.15.58:0/818981686 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fa85402fcc0 msgr2=0x7fa854032180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:24.684 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.682+0000 7fa87979b700 1 --2- 172.21.15.58:0/818981686 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fa85402fcc0 0x7fa854032180 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa85c0096a0 tx=0x7fa85c006b40).stop 2021-11-20T05:29:24.684 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.682+0000 7fa87979b700 1 -- 172.21.15.58:0/818981686 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa87406fcc0 msgr2=0x7fa87410cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:24.684 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.682+0000 7fa87979b700 1 --2- 172.21.15.58:0/818981686 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa87406fcc0 0x7fa87410cd20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fa86c006dc0 tx=0x7fa86c0092d0).stop 2021-11-20T05:29:24.684 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.683+0000 7fa87979b700 1 -- 172.21.15.58:0/818981686 shutdown_connections 2021-11-20T05:29:24.685 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.683+0000 7fa87979b700 1 --2- 172.21.15.58:0/818981686 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fa85402fcc0 0x7fa854032180 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:24.685 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.683+0000 7fa87979b700 1 --2- 172.21.15.58:0/818981686 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa87406fcc0 0x7fa87410cd20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:24.685 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.683+0000 7fa87979b700 1 -- 172.21.15.58:0/818981686 >> 172.21.15.58:0/818981686 conn(0x7fa8740f62e0 msgr2=0x7fa8740f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:24.685 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.683+0000 7fa87979b700 1 -- 172.21.15.58:0/818981686 shutdown_connections 2021-11-20T05:29:24.686 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:24.683+0000 7fa87979b700 1 -- 172.21.15.58:0/818981686 wait complete. 2021-11-20T05:29:25.026 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2021-11-20T05:29:25.554 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.551+0000 7f659fb63700 1 Processor -- start 2021-11-20T05:29:25.554 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.551+0000 7f659fb63700 1 -- start start 2021-11-20T05:29:25.555 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.551+0000 7f659fb63700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65980fd2e0 0x7f65980fd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:25.555 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.551+0000 7f659fb63700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65980fdbf0 con 0x7f65980fd2e0 2021-11-20T05:29:25.555 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.551+0000 7f659d8ff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65980fd2e0 0x7f65980fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:25.555 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.551+0000 7f659d8ff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65980fd2e0 0x7f65980fd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39616/0 (socket says 172.21.15.58:39616) 2021-11-20T05:29:25.555 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.551+0000 7f659d8ff700 1 -- 172.21.15.58:0/702576314 learned_addr learned my addr 172.21.15.58:0/702576314 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:25.556 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.552+0000 7f659d8ff700 1 -- 172.21.15.58:0/702576314 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65980fdd30 con 0x7f65980fd2e0 2021-11-20T05:29:25.556 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.552+0000 7f659d8ff700 1 --2- 172.21.15.58:0/702576314 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65980fd2e0 0x7f65980fd700 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f659400b5f0 tx=0x7f6594006ee0).ready entity=mon.0 client_cookie=652c4b8c69782a7d server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:25.556 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.552+0000 7f659c8fd700 1 -- 172.21.15.58:0/702576314 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f659400d600 con 0x7f65980fd2e0 2021-11-20T05:29:25.556 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.552+0000 7f659c8fd700 1 -- 172.21.15.58:0/702576314 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f659400dbd0 con 0x7f65980fd2e0 2021-11-20T05:29:25.557 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.552+0000 7f659c8fd700 1 -- 172.21.15.58:0/702576314 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6594011c00 con 0x7f65980fd2e0 2021-11-20T05:29:25.557 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.552+0000 7f659fb63700 1 -- 172.21.15.58:0/702576314 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65980fd2e0 msgr2=0x7f65980fd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:25.557 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.552+0000 7f659fb63700 1 --2- 172.21.15.58:0/702576314 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65980fd2e0 0x7f65980fd700 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f659400b5f0 tx=0x7f6594006ee0).stop 2021-11-20T05:29:25.557 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.552+0000 7f659fb63700 1 -- 172.21.15.58:0/702576314 shutdown_connections 2021-11-20T05:29:25.557 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.552+0000 7f659fb63700 1 --2- 172.21.15.58:0/702576314 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65980fd2e0 0x7f65980fd700 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:25.558 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.552+0000 7f659fb63700 1 -- 172.21.15.58:0/702576314 >> 172.21.15.58:0/702576314 conn(0x7f65980f5a80 msgr2=0x7f65980f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:25.558 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.553+0000 7f659fb63700 1 -- 172.21.15.58:0/702576314 shutdown_connections 2021-11-20T05:29:25.559 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.553+0000 7f659fb63700 1 -- 172.21.15.58:0/702576314 wait complete. 2021-11-20T05:29:25.559 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.553+0000 7f659fb63700 1 Processor -- start 2021-11-20T05:29:25.559 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.553+0000 7f659fb63700 1 -- start start 2021-11-20T05:29:25.559 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.554+0000 7f659fb63700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65980fd2e0 0x7f659810a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:25.560 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.554+0000 7f659fb63700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659810ae90 con 0x7f65980fd2e0 2021-11-20T05:29:25.560 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.554+0000 7f659d8ff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65980fd2e0 0x7f659810a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:25.560 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.554+0000 7f659d8ff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65980fd2e0 0x7f659810a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39618/0 (socket says 172.21.15.58:39618) 2021-11-20T05:29:25.560 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.554+0000 7f659d8ff700 1 -- 172.21.15.58:0/3281557574 learned_addr learned my addr 172.21.15.58:0/3281557574 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:25.561 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.554+0000 7f659d8ff700 1 -- 172.21.15.58:0/3281557574 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659400b200 con 0x7f65980fd2e0 2021-11-20T05:29:25.561 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.554+0000 7f659d8ff700 1 --2- 172.21.15.58:0/3281557574 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65980fd2e0 0x7f659810a9a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f659400e040 tx=0x7f6594006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:25.561 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.554+0000 7f658a7fc700 1 -- 172.21.15.58:0/3281557574 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f659401b410 con 0x7f65980fd2e0 2021-11-20T05:29:25.561 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.554+0000 7f658a7fc700 1 -- 172.21.15.58:0/3281557574 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f659401b9e0 con 0x7f65980fd2e0 2021-11-20T05:29:25.561 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.554+0000 7f658a7fc700 1 -- 172.21.15.58:0/3281557574 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f659401a770 con 0x7f65980fd2e0 2021-11-20T05:29:25.563 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.555+0000 7f659fb63700 1 -- 172.21.15.58:0/3281557574 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659810b090 con 0x7f65980fd2e0 2021-11-20T05:29:25.564 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.555+0000 7f659fb63700 1 -- 172.21.15.58:0/3281557574 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659810b4a0 con 0x7f65980fd2e0 2021-11-20T05:29:25.564 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.556+0000 7f658a7fc700 1 -- 172.21.15.58:0/3281557574 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f659400f070 con 0x7f65980fd2e0 2021-11-20T05:29:25.564 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.556+0000 7f658a7fc700 1 --2- 172.21.15.58:0/3281557574 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f658002fdb0 0x7f6580032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:25.564 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.556+0000 7f658a7fc700 1 -- 172.21.15.58:0/3281557574 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f6594040b00 con 0x7f65980fd2e0 2021-11-20T05:29:25.565 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.556+0000 7f659d0fe700 1 --2- 172.21.15.58:0/3281557574 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f658002fdb0 0x7f6580032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:25.565 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.556+0000 7f659fb63700 1 -- 172.21.15.58:0/3281557574 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6598059100 con 0x7f65980fd2e0 2021-11-20T05:29:25.565 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.557+0000 7f659d0fe700 1 --2- 172.21.15.58:0/3281557574 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f658002fdb0 0x7f6580032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f658c0096a0 tx=0x7f658c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:25.565 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.561+0000 7f658a7fc700 1 -- 172.21.15.58:0/3281557574 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6594016020 con 0x7f65980fd2e0 2021-11-20T05:29:25.851 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.851+0000 7f659fb63700 1 -- 172.21.15.58:0/3281557574 --> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f6598104590 con 0x7f658002fdb0 2021-11-20T05:29:25.852 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.852+0000 7f658a7fc700 1 -- 172.21.15.58:0/3281557574 <== mgr.14116 v2:172.21.15.58:6800/1261811851 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f6598104590 con 0x7f658002fdb0 2021-11-20T05:29:25.852 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: value unchanged 2021-11-20T05:29:25.855 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.854+0000 7f659fb63700 1 -- 172.21.15.58:0/3281557574 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f658002fdb0 msgr2=0x7f6580032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:25.855 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.854+0000 7f659fb63700 1 --2- 172.21.15.58:0/3281557574 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f658002fdb0 0x7f6580032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f658c0096a0 tx=0x7f658c006b40).stop 2021-11-20T05:29:25.856 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.854+0000 7f659fb63700 1 -- 172.21.15.58:0/3281557574 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65980fd2e0 msgr2=0x7f659810a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:25.856 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.854+0000 7f659fb63700 1 --2- 172.21.15.58:0/3281557574 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65980fd2e0 0x7f659810a9a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f659400e040 tx=0x7f6594006ee0).stop 2021-11-20T05:29:25.856 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.854+0000 7f659fb63700 1 -- 172.21.15.58:0/3281557574 shutdown_connections 2021-11-20T05:29:25.857 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.854+0000 7f659fb63700 1 --2- 172.21.15.58:0/3281557574 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f658002fdb0 0x7f6580032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:25.857 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.854+0000 7f659fb63700 1 --2- 172.21.15.58:0/3281557574 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65980fd2e0 0x7f659810a9a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:25.857 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.854+0000 7f659fb63700 1 -- 172.21.15.58:0/3281557574 >> 172.21.15.58:0/3281557574 conn(0x7f65980f5a80 msgr2=0x7f65980ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:25.857 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.854+0000 7f659fb63700 1 -- 172.21.15.58:0/3281557574 shutdown_connections 2021-11-20T05:29:25.858 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:25.854+0000 7f659fb63700 1 -- 172.21.15.58:0/3281557574 wait complete. 2021-11-20T05:29:25.969 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:25 smithi058 conmon[33537]: audit 2021-11-20T05:29:24.677240+0000 mgr.smithi058.zagdlk (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:29:25.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:25 smithi058 conmon[33537]: audit 2021-11-20T05:29:24.681290+0000 mon.smithi058 (mon.0) 50 : audit [INF] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:25.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:25 smithi058 conmon[33537]: audit 2021-11-20T05:29:25.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:25 smithi058 conmon[33537]: 2021-11 2021-11-20T05:29:25.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:25 smithi058 conmon[33537]: -20T05:29:24.686044+0000 2021-11-20T05:29:25.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:25 smithi058 conmon[33537]: mon.smithi058 (mon.0) 51 : audit [DBG] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:29:25.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:25 smithi058 conmon[33537]: cluster 2021-11 2021-11-20T05:29:25.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:25 smithi058 conmon[33537]: -20T05:29:25.161455+0000 mon.smithi058 (mon.0 2021-11-20T05:29:25.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:25 smithi058 conmon[33537]: ) 52 : cluster [DBG] mgrmap e8: smithi058.zagdlk(active, since 2s) 2021-11-20T05:29:26.262 INFO:teuthology.orchestra.run.smithi058.stderr:Generating ssh key... 2021-11-20T05:29:26.263 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2021-11-20T05:29:26.724 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.721+0000 7f963c919700 1 Processor -- start 2021-11-20T05:29:26.725 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.721+0000 7f963c919700 1 -- start start 2021-11-20T05:29:26.725 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.721+0000 7f963c919700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9638071cd0 0x7f9638070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:26.725 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.721+0000 7f963c919700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96380720f0 con 0x7f9638071cd0 2021-11-20T05:29:26.725 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.721+0000 7f963659c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9638071cd0 0x7f9638070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:26.726 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.721+0000 7f963659c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9638071cd0 0x7f9638070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39622/0 (socket says 172.21.15.58:39622) 2021-11-20T05:29:26.726 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.721+0000 7f963659c700 1 -- 172.21.15.58:0/1548663394 learned_addr learned my addr 172.21.15.58:0/1548663394 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:26.726 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.722+0000 7f963659c700 1 -- 172.21.15.58:0/1548663394 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9638070760 con 0x7f9638071cd0 2021-11-20T05:29:26.726 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.722+0000 7f963659c700 1 --2- 172.21.15.58:0/1548663394 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9638071cd0 0x7f9638070270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f962800b5f0 tx=0x7f9628006ee0).ready entity=mon.0 client_cookie=c91ba89fc94bbb61 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:26.727 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.722+0000 7f963559a700 1 -- 172.21.15.58:0/1548663394 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f962800d600 con 0x7f9638071cd0 2021-11-20T05:29:26.727 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.722+0000 7f963559a700 1 -- 172.21.15.58:0/1548663394 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f962800dbd0 con 0x7f9638071cd0 2021-11-20T05:29:26.727 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.722+0000 7f963559a700 1 -- 172.21.15.58:0/1548663394 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9628011c00 con 0x7f9638071cd0 2021-11-20T05:29:26.727 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.722+0000 7f963c919700 1 -- 172.21.15.58:0/1548663394 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9638071cd0 msgr2=0x7f9638070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:26.728 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.722+0000 7f963c919700 1 --2- 172.21.15.58:0/1548663394 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9638071cd0 0x7f9638070270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f962800b5f0 tx=0x7f9628006ee0).stop 2021-11-20T05:29:26.728 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.723+0000 7f963c919700 1 -- 172.21.15.58:0/1548663394 shutdown_connections 2021-11-20T05:29:26.728 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.723+0000 7f963c919700 1 --2- 172.21.15.58:0/1548663394 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9638071cd0 0x7f9638070270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:26.728 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.723+0000 7f963c919700 1 -- 172.21.15.58:0/1548663394 >> 172.21.15.58:0/1548663394 conn(0x7f96380f73e0 msgr2=0x7f96380f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:26.729 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.723+0000 7f963c919700 1 -- 172.21.15.58:0/1548663394 shutdown_connections 2021-11-20T05:29:26.729 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.723+0000 7f963c919700 1 -- 172.21.15.58:0/1548663394 wait complete. 2021-11-20T05:29:26.729 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.723+0000 7f963c919700 1 Processor -- start 2021-11-20T05:29:26.729 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.723+0000 7f963c919700 1 -- start start 2021-11-20T05:29:26.729 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.723+0000 7f963c919700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9638071cd0 0x7f963810cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:26.730 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.723+0000 7f963c919700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963810d040 con 0x7f9638071cd0 2021-11-20T05:29:26.730 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.724+0000 7f963659c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9638071cd0 0x7f963810cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:26.730 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.724+0000 7f963659c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9638071cd0 0x7f963810cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39624/0 (socket says 172.21.15.58:39624) 2021-11-20T05:29:26.730 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.724+0000 7f963659c700 1 -- 172.21.15.58:0/3137353048 learned_addr learned my addr 172.21.15.58:0/3137353048 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:26.731 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.724+0000 7f963659c700 1 -- 172.21.15.58:0/3137353048 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962800b200 con 0x7f9638071cd0 2021-11-20T05:29:26.731 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.724+0000 7f963659c700 1 --2- 172.21.15.58:0/3137353048 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9638071cd0 0x7f963810cb50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f962800b190 tx=0x7f9628006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:26.731 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.724+0000 7f9626ffd700 1 -- 172.21.15.58:0/3137353048 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f962801b410 con 0x7f9638071cd0 2021-11-20T05:29:26.732 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.724+0000 7f9626ffd700 1 -- 172.21.15.58:0/3137353048 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f962801b9e0 con 0x7f9638071cd0 2021-11-20T05:29:26.732 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.724+0000 7f963c919700 1 -- 172.21.15.58:0/3137353048 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963810d240 con 0x7f9638071cd0 2021-11-20T05:29:26.732 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.724+0000 7f9626ffd700 1 -- 172.21.15.58:0/3137353048 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f962801a880 con 0x7f9638071cd0 2021-11-20T05:29:26.732 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.724+0000 7f963c919700 1 -- 172.21.15.58:0/3137353048 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f963810d650 con 0x7f9638071cd0 2021-11-20T05:29:26.732 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.725+0000 7f9626ffd700 1 -- 172.21.15.58:0/3137353048 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f962800f070 con 0x7f9638071cd0 2021-11-20T05:29:26.734 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.725+0000 7f9626ffd700 1 --2- 172.21.15.58:0/3137353048 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f961c02fdb0 0x7f961c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:26.734 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.725+0000 7f9626ffd700 1 -- 172.21.15.58:0/3137353048 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9628041170 con 0x7f9638071cd0 2021-11-20T05:29:26.734 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.725+0000 7f9635d9b700 1 --2- 172.21.15.58:0/3137353048 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f961c02fdb0 0x7f961c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:26.735 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.726+0000 7f963c919700 1 -- 172.21.15.58:0/3137353048 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9638059100 con 0x7f9638071cd0 2021-11-20T05:29:26.735 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.726+0000 7f9635d9b700 1 --2- 172.21.15.58:0/3137353048 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f961c02fdb0 0x7f961c032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f962c0096a0 tx=0x7f962c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:26.735 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:26.730+0000 7f9626ffd700 1 -- 172.21.15.58:0/3137353048 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9628016020 con 0x7f9638071cd0 2021-11-20T05:29:26.969 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:26 smithi058 conmon[33537]: audit 2021-11-20T05:29:25.852887+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:29:26.969 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:26 smithi058 conmon[33537]: mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:29:27.019 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:27.019+0000 7f963c919700 1 -- 172.21.15.58:0/3137353048 --> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f9638106a50 con 0x7f961c02fdb0 2021-11-20T05:29:27.306 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:27.307+0000 7f9626ffd700 1 -- 172.21.15.58:0/3137353048 <== mgr.14116 v2:172.21.15.58:6800/1261811851 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9638106a50 con 0x7f961c02fdb0 2021-11-20T05:29:27.309 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:27.308+0000 7f963c919700 1 -- 172.21.15.58:0/3137353048 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f961c02fdb0 msgr2=0x7f961c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:27.309 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:27.308+0000 7f963c919700 1 --2- 172.21.15.58:0/3137353048 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f961c02fdb0 0x7f961c032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f962c0096a0 tx=0x7f962c006b40).stop 2021-11-20T05:29:27.309 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:27.308+0000 7f963c919700 1 -- 172.21.15.58:0/3137353048 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9638071cd0 msgr2=0x7f963810cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:27.310 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:27.308+0000 7f963c919700 1 --2- 172.21.15.58:0/3137353048 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9638071cd0 0x7f963810cb50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f962800b190 tx=0x7f9628006ee0).stop 2021-11-20T05:29:27.310 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:27.308+0000 7f963c919700 1 -- 172.21.15.58:0/3137353048 shutdown_connections 2021-11-20T05:29:27.310 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:27.308+0000 7f963c919700 1 --2- 172.21.15.58:0/3137353048 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f961c02fdb0 0x7f961c032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:27.310 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:27.308+0000 7f963c919700 1 --2- 172.21.15.58:0/3137353048 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9638071cd0 0x7f963810cb50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:27.310 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:27.308+0000 7f963c919700 1 -- 172.21.15.58:0/3137353048 >> 172.21.15.58:0/3137353048 conn(0x7f96380f73e0 msgr2=0x7f96380f8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:27.311 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:27.309+0000 7f963c919700 1 -- 172.21.15.58:0/3137353048 shutdown_connections 2021-11-20T05:29:27.311 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:27.309+0000 7f963c919700 1 -- 172.21.15.58:0/3137353048 wait complete. 2021-11-20T05:29:27.700 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2021-11-20T05:29:28.209 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.206+0000 7f9de7a94700 1 Processor -- start 2021-11-20T05:29:28.209 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.206+0000 7f9de7a94700 1 -- start start 2021-11-20T05:29:28.210 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.207+0000 7f9de7a94700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9de00fd200 0x7f9de00fd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:28.210 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.207+0000 7f9de7a94700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de00fdb10 con 0x7f9de00fd200 2021-11-20T05:29:28.210 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.207+0000 7f9de5830700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9de00fd200 0x7f9de00fd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:28.211 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.207+0000 7f9de5830700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9de00fd200 0x7f9de00fd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39628/0 (socket says 172.21.15.58:39628) 2021-11-20T05:29:28.211 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.207+0000 7f9de5830700 1 -- 172.21.15.58:0/1106015675 learned_addr learned my addr 172.21.15.58:0/1106015675 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:28.211 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.207+0000 7f9de5830700 1 -- 172.21.15.58:0/1106015675 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de00fdc50 con 0x7f9de00fd200 2021-11-20T05:29:28.211 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.207+0000 7f9de5830700 1 --2- 172.21.15.58:0/1106015675 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9de00fd200 0x7f9de00fd620 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9dd000b5f0 tx=0x7f9dd0006ee0).ready entity=mon.0 client_cookie=1cafa301ab2f83a8 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:28.211 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.207+0000 7f9de482e700 1 -- 172.21.15.58:0/1106015675 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dd000d600 con 0x7f9de00fd200 2021-11-20T05:29:28.212 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.207+0000 7f9de482e700 1 -- 172.21.15.58:0/1106015675 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9dd000dbd0 con 0x7f9de00fd200 2021-11-20T05:29:28.212 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.208+0000 7f9de482e700 1 -- 172.21.15.58:0/1106015675 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dd0011c00 con 0x7f9de00fd200 2021-11-20T05:29:28.212 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.208+0000 7f9de7a94700 1 -- 172.21.15.58:0/1106015675 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9de00fd200 msgr2=0x7f9de00fd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:28.212 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.208+0000 7f9de7a94700 1 --2- 172.21.15.58:0/1106015675 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9de00fd200 0x7f9de00fd620 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9dd000b5f0 tx=0x7f9dd0006ee0).stop 2021-11-20T05:29:28.213 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.208+0000 7f9de7a94700 1 -- 172.21.15.58:0/1106015675 shutdown_connections 2021-11-20T05:29:28.213 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.208+0000 7f9de7a94700 1 --2- 172.21.15.58:0/1106015675 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9de00fd200 0x7f9de00fd620 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:28.213 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.208+0000 7f9de7a94700 1 -- 172.21.15.58:0/1106015675 >> 172.21.15.58:0/1106015675 conn(0x7f9de00f5a80 msgr2=0x7f9de00f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:28.213 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.208+0000 7f9de7a94700 1 -- 172.21.15.58:0/1106015675 shutdown_connections 2021-11-20T05:29:28.213 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.208+0000 7f9de7a94700 1 -- 172.21.15.58:0/1106015675 wait complete. 2021-11-20T05:29:28.214 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.209+0000 7f9de7a94700 1 Processor -- start 2021-11-20T05:29:28.214 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.209+0000 7f9de7a94700 1 -- start start 2021-11-20T05:29:28.214 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.209+0000 7f9de7a94700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9de00fd200 0x7f9de010a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:28.214 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.209+0000 7f9de7a94700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de010aed0 con 0x7f9de00fd200 2021-11-20T05:29:28.215 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.209+0000 7f9de5830700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9de00fd200 0x7f9de010a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:28.215 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.209+0000 7f9de5830700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9de00fd200 0x7f9de010a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39630/0 (socket says 172.21.15.58:39630) 2021-11-20T05:29:28.215 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.209+0000 7f9de5830700 1 -- 172.21.15.58:0/2516625147 learned_addr learned my addr 172.21.15.58:0/2516625147 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:28.217 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.210+0000 7f9de5830700 1 -- 172.21.15.58:0/2516625147 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd000b200 con 0x7f9de00fd200 2021-11-20T05:29:28.218 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.210+0000 7f9de5830700 1 --2- 172.21.15.58:0/2516625147 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9de00fd200 0x7f9de010a9e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f9dd000e040 tx=0x7f9dd0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:28.218 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.210+0000 7f9dce7fc700 1 -- 172.21.15.58:0/2516625147 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dd001b410 con 0x7f9de00fd200 2021-11-20T05:29:28.218 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.210+0000 7f9de7a94700 1 -- 172.21.15.58:0/2516625147 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de010b0d0 con 0x7f9de00fd200 2021-11-20T05:29:28.218 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.210+0000 7f9dce7fc700 1 -- 172.21.15.58:0/2516625147 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9dd001b9e0 con 0x7f9de00fd200 2021-11-20T05:29:28.218 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.210+0000 7f9dce7fc700 1 -- 172.21.15.58:0/2516625147 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dd001a770 con 0x7f9de00fd200 2021-11-20T05:29:28.220 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.210+0000 7f9de7a94700 1 -- 172.21.15.58:0/2516625147 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de010b4e0 con 0x7f9de00fd200 2021-11-20T05:29:28.221 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.211+0000 7f9dce7fc700 1 -- 172.21.15.58:0/2516625147 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9dd000f070 con 0x7f9de00fd200 2021-11-20T05:29:28.221 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.211+0000 7f9dce7fc700 1 --2- 172.21.15.58:0/2516625147 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f9dc802fd50 0x7f9dc8032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:28.221 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.211+0000 7f9dce7fc700 1 -- 172.21.15.58:0/2516625147 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9dd0041050 con 0x7f9de00fd200 2021-11-20T05:29:28.221 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.211+0000 7f9de502f700 1 --2- 172.21.15.58:0/2516625147 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f9dc802fd50 0x7f9dc8032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:28.222 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.211+0000 7f9de7a94700 1 -- 172.21.15.58:0/2516625147 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de0045ad0 con 0x7f9de00fd200 2021-11-20T05:29:28.222 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.211+0000 7f9de502f700 1 --2- 172.21.15.58:0/2516625147 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f9dc802fd50 0x7f9dc8032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9ddc0096a0 tx=0x7f9ddc006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:28.222 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.215+0000 7f9dce7fc700 1 -- 172.21.15.58:0/2516625147 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9dd0024410 con 0x7f9de00fd200 2021-11-20T05:29:28.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:28 smithi058 conmon[33537]: cephadm 2021-11-20T05:29:27.020796+0000 mgr.smithi058.zagdlk (mgr.14116) 6 : cephadm 2021-11-20T05:29:28.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:28 smithi058 conmon[33537]: [INF] Generating ssh key... 2021-11-20T05:29:28.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:28 smithi058 conmon[33537]: audit 2021-11-20T05:29:27.305105+0000 mon.smithi058 2021-11-20T05:29:28.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:28 smithi058 conmon[33537]: (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:28.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:28 smithi058 conmon[33537]: audit 2021 2021-11-20T05:29:28.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:28 smithi058 conmon[33537]: -11-20T05:29:27 2021-11-20T05:29:28.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:28 smithi058 conmon[33537]: .306790+0000 mon.smithi058 (mon. 2021-11-20T05:29:28.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:28 smithi058 conmon[33537]: 0) 54 : audit [INF] 2021-11-20T05:29:28.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:28 smithi058 conmon[33537]: from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:28.508 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.508+0000 7f9de7a94700 1 -- 172.21.15.58:0/2516625147 --> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f9de01045c0 con 0x7f9dc802fd50 2021-11-20T05:29:28.508 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.508+0000 7f9dce7fc700 1 -- 172.21.15.58:0/2516625147 <== mgr.14116 v2:172.21.15.58:6800/1261811851 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f9de01045c0 con 0x7f9dc802fd50 2021-11-20T05:29:28.510 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC86XmLpBDHobQdn3NGWif/qFSOW3QeM6e79vZ/ZCb2YHk00xy4YVhc+XXrE3Spq4TALoFcEge2K0yPCQVITrzG8kYfkUQiEI2J8iTh7PGOVonOWSNbPCldnjXyJJK0NYloqunrKaBlbCyjsJMQpGV4GOx6zvzWtxnPN9bvmEMDqSSF0HdcF+oMJf31bcjFJRqcYy1tUId1uf4RdfN1nCEcyrcHipSgT8ki+GoShA8faayD/v3PhmverG2YSNyxvWfQ5bfK9vPncOPs0wukrY39EKdy7I7CmpMCWGigy9+0zYBXAeW/4lpiHsy6FKp/WLuRkidsZm4+ohTIJ8jba9vJD1Ko6x67pV86qKmtnt889QtakunwswYAkmb5Z9oE7mG5AuSiDqRxR93uyf1fU0EiHdJwx9r/8Lnh3s+N2UvevjNatLT1yVO7BvuDbf+MvIhqnj6CH1Qrj+4kWEJCzskb/m9d0bxSR2ZL1PHbVTXGz5VgWcVMGqsIdq4yhVypl2k= ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:29:28.512 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.510+0000 7f9de7a94700 1 -- 172.21.15.58:0/2516625147 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f9dc802fd50 msgr2=0x7f9dc8032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:28.512 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.510+0000 7f9de7a94700 1 --2- 172.21.15.58:0/2516625147 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f9dc802fd50 0x7f9dc8032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9ddc0096a0 tx=0x7f9ddc006b40).stop 2021-11-20T05:29:28.513 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.510+0000 7f9de7a94700 1 -- 172.21.15.58:0/2516625147 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9de00fd200 msgr2=0x7f9de010a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:28.513 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.510+0000 7f9de7a94700 1 --2- 172.21.15.58:0/2516625147 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9de00fd200 0x7f9de010a9e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f9dd000e040 tx=0x7f9dd0006ee0).stop 2021-11-20T05:29:28.513 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.510+0000 7f9de7a94700 1 -- 172.21.15.58:0/2516625147 shutdown_connections 2021-11-20T05:29:28.513 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.510+0000 7f9de7a94700 1 --2- 172.21.15.58:0/2516625147 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f9dc802fd50 0x7f9dc8032210 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:28.514 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.510+0000 7f9de7a94700 1 --2- 172.21.15.58:0/2516625147 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9de00fd200 0x7f9de010a9e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:28.514 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.510+0000 7f9de7a94700 1 -- 172.21.15.58:0/2516625147 >> 172.21.15.58:0/2516625147 conn(0x7f9de00f5a80 msgr2=0x7f9de00ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:28.515 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.511+0000 7f9de7a94700 1 -- 172.21.15.58:0/2516625147 shutdown_connections 2021-11-20T05:29:28.517 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:28.511+0000 7f9de7a94700 1 -- 172.21.15.58:0/2516625147 wait complete. 2021-11-20T05:29:28.839 INFO:teuthology.orchestra.run.smithi058.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2021-11-20T05:29:28.840 INFO:teuthology.orchestra.run.smithi058.stderr:Adding key to root@localhost authorized_keys... 2021-11-20T05:29:28.840 INFO:teuthology.orchestra.run.smithi058.stderr:Adding host smithi058... 2021-11-20T05:29:28.841 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi058 172.21.15.58 2021-11-20T05:29:29.330 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.327+0000 7f05a9ab3700 1 Processor -- start 2021-11-20T05:29:29.439 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.327+0000 7f05a9ab3700 1 -- start start 2021-11-20T05:29:29.440 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.327+0000 7f05a9ab3700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f05a40fa020 0x7f05a40fa440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:29.440 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.327+0000 7f05a9ab3700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a40fa930 con 0x7f05a40fa020 2021-11-20T05:29:29.440 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.327+0000 7f05a37fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f05a40fa020 0x7f05a40fa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:29.440 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.328+0000 7f05a37fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f05a40fa020 0x7f05a40fa440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39634/0 (socket says 172.21.15.58:39634) 2021-11-20T05:29:29.441 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.328+0000 7f05a37fe700 1 -- 172.21.15.58:0/3115147381 learned_addr learned my addr 172.21.15.58:0/3115147381 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:29.441 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.328+0000 7f05a37fe700 1 -- 172.21.15.58:0/3115147381 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a40faa70 con 0x7f05a40fa020 2021-11-20T05:29:29.442 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.328+0000 7f05a37fe700 1 --2- 172.21.15.58:0/3115147381 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f05a40fa020 0x7f05a40fa440 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f059400b5f0 tx=0x7f0594006ee0).ready entity=mon.0 client_cookie=5f8700813249d285 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:29.442 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.328+0000 7f05a27fc700 1 -- 172.21.15.58:0/3115147381 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f059400d600 con 0x7f05a40fa020 2021-11-20T05:29:29.442 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.328+0000 7f05a27fc700 1 -- 172.21.15.58:0/3115147381 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f059400dbd0 con 0x7f05a40fa020 2021-11-20T05:29:29.443 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.328+0000 7f05a27fc700 1 -- 172.21.15.58:0/3115147381 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0594011c00 con 0x7f05a40fa020 2021-11-20T05:29:29.443 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.329+0000 7f05a9ab3700 1 -- 172.21.15.58:0/3115147381 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f05a40fa020 msgr2=0x7f05a40fa440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:29.443 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.329+0000 7f05a9ab3700 1 --2- 172.21.15.58:0/3115147381 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f05a40fa020 0x7f05a40fa440 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f059400b5f0 tx=0x7f0594006ee0).stop 2021-11-20T05:29:29.444 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.329+0000 7f05a9ab3700 1 -- 172.21.15.58:0/3115147381 shutdown_connections 2021-11-20T05:29:29.444 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.329+0000 7f05a9ab3700 1 --2- 172.21.15.58:0/3115147381 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f05a40fa020 0x7f05a40fa440 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:29.444 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.329+0000 7f05a9ab3700 1 -- 172.21.15.58:0/3115147381 >> 172.21.15.58:0/3115147381 conn(0x7f05a40f5a80 msgr2=0x7f05a40f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:29.444 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.329+0000 7f05a9ab3700 1 -- 172.21.15.58:0/3115147381 shutdown_connections 2021-11-20T05:29:29.445 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.329+0000 7f05a9ab3700 1 -- 172.21.15.58:0/3115147381 wait complete. 2021-11-20T05:29:29.445 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.329+0000 7f05a9ab3700 1 Processor -- start 2021-11-20T05:29:29.445 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.330+0000 7f05a9ab3700 1 -- start start 2021-11-20T05:29:29.445 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.330+0000 7f05a9ab3700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f05a40fcbd0 0x7f05a40fb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:29.445 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.330+0000 7f05a9ab3700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a40fcff0 con 0x7f05a40fcbd0 2021-11-20T05:29:29.446 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.330+0000 7f05a37fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f05a40fcbd0 0x7f05a40fb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:29.446 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.330+0000 7f05a37fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f05a40fcbd0 0x7f05a40fb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39636/0 (socket says 172.21.15.58:39636) 2021-11-20T05:29:29.446 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.330+0000 7f05a37fe700 1 -- 172.21.15.58:0/368162664 learned_addr learned my addr 172.21.15.58:0/368162664 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:29.446 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.330+0000 7f05a37fe700 1 -- 172.21.15.58:0/368162664 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059400b200 con 0x7f05a40fcbd0 2021-11-20T05:29:29.447 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.330+0000 7f05a37fe700 1 --2- 172.21.15.58:0/368162664 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f05a40fcbd0 0x7f05a40fb090 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f059400b5c0 tx=0x7f0594006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:29.448 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.331+0000 7f058bfff700 1 -- 172.21.15.58:0/368162664 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0594011a20 con 0x7f05a40fcbd0 2021-11-20T05:29:29.448 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.331+0000 7f05a9ab3700 1 -- 172.21.15.58:0/368162664 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a40fb580 con 0x7f05a40fcbd0 2021-11-20T05:29:29.449 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.331+0000 7f05a9ab3700 1 -- 172.21.15.58:0/368162664 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a40fb9f0 con 0x7f05a40fcbd0 2021-11-20T05:29:29.449 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.331+0000 7f058bfff700 1 -- 172.21.15.58:0/368162664 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f05940258d0 con 0x7f05a40fcbd0 2021-11-20T05:29:29.449 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.331+0000 7f058bfff700 1 -- 172.21.15.58:0/368162664 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f059401b680 con 0x7f05a40fcbd0 2021-11-20T05:29:29.449 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.331+0000 7f058bfff700 1 -- 172.21.15.58:0/368162664 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f059400f070 con 0x7f05a40fcbd0 2021-11-20T05:29:29.450 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.332+0000 7f058bfff700 1 --2- 172.21.15.58:0/368162664 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f059002fd50 0x7f0590032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:29.450 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.332+0000 7f058bfff700 1 -- 172.21.15.58:0/368162664 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0594041f60 con 0x7f05a40fcbd0 2021-11-20T05:29:29.450 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.332+0000 7f05a2ffd700 1 --2- 172.21.15.58:0/368162664 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f059002fd50 0x7f0590032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:29.451 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.332+0000 7f05a9ab3700 1 -- 172.21.15.58:0/368162664 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0584004fa0 con 0x7f05a40fcbd0 2021-11-20T05:29:29.451 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.332+0000 7f05a2ffd700 1 --2- 172.21.15.58:0/368162664 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f059002fd50 0x7f0590032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f05980096a0 tx=0x7f0598006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:29.452 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.336+0000 7f058bfff700 1 -- 172.21.15.58:0/368162664 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0594016070 con 0x7f05a40fcbd0 2021-11-20T05:29:29.665 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:29.665+0000 7f05a9ab3700 1 -- 172.21.15.58:0/368162664 --> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi058", "addr": "172.21.15.58", "target": ["mon-mgr", ""]}) v1 -- 0x7f05840029f0 con 0x7f059002fd50 2021-11-20T05:29:30.383 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: Added host 'smithi058' 2021-11-20T05:29:30.383 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:30.381+0000 7f058bfff700 1 -- 172.21.15.58:0/368162664 <== mgr.14116 v2:172.21.15.58:6800/1261811851 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f05840029f0 con 0x7f059002fd50 2021-11-20T05:29:30.384 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:30.384+0000 7f05a9ab3700 1 -- 172.21.15.58:0/368162664 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f059002fd50 msgr2=0x7f0590032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:30.384 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:30.384+0000 7f05a9ab3700 1 --2- 172.21.15.58:0/368162664 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f059002fd50 0x7f0590032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f05980096a0 tx=0x7f0598006b40).stop 2021-11-20T05:29:30.384 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:30.384+0000 7f05a9ab3700 1 -- 172.21.15.58:0/368162664 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f05a40fcbd0 msgr2=0x7f05a40fb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:30.385 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:30.384+0000 7f05a9ab3700 1 --2- 172.21.15.58:0/368162664 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f05a40fcbd0 0x7f05a40fb090 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f059400b5c0 tx=0x7f0594006ee0).stop 2021-11-20T05:29:30.385 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:30.385+0000 7f05a9ab3700 1 -- 172.21.15.58:0/368162664 shutdown_connections 2021-11-20T05:29:30.385 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:30.385+0000 7f05a9ab3700 1 --2- 172.21.15.58:0/368162664 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f059002fd50 0x7f0590032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:30.385 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:30.385+0000 7f05a9ab3700 1 --2- 172.21.15.58:0/368162664 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f05a40fcbd0 0x7f05a40fb090 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:30.386 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:30.385+0000 7f05a9ab3700 1 -- 172.21.15.58:0/368162664 >> 172.21.15.58:0/368162664 conn(0x7f05a40f5a80 msgr2=0x7f05a40f76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:30.386 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:30.385+0000 7f05a9ab3700 1 -- 172.21.15.58:0/368162664 shutdown_connections 2021-11-20T05:29:30.386 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:30.385+0000 7f05a9ab3700 1 -- 172.21.15.58:0/368162664 wait complete. 2021-11-20T05:29:30.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:30 smithi058 conmon[33537]: audit 2021-11-20T05:29:28.509525+0000 mgr.smithi058.zagdlk (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:29:30.781 INFO:teuthology.orchestra.run.smithi058.stderr:Deploying mon service with default placement... 2021-11-20T05:29:30.782 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2021-11-20T05:29:31.219 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:31 smithi058 conmon[33537]: audit 2021-11-20T05:29:29.667209+0000 mgr.smithi058.zagdlk (mgr.14116) 8 : 2021-11-20T05:29:31.219 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:31 smithi058 conmon[33537]: audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi058", "addr": "172.21.15.58", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:29:31.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:31 smithi058 conmon[33537]: audit 2021-11-20T05:29:30.381944+0000 mon.smithi058 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:31.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:31 smithi058 conmon[33537]: audit 2021-11-20T05:29:30.382504+0000 mon.smithi058 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:29:31.277 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.273+0000 7f02222bd700 1 Processor -- start 2021-11-20T05:29:31.277 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.273+0000 7f02222bd700 1 -- start start 2021-11-20T05:29:31.278 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.273+0000 7f02222bd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0fbe90 0x7f021c0fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:31.278 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.274+0000 7f02222bd700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c0fc7a0 con 0x7f021c0fbe90 2021-11-20T05:29:31.278 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.274+0000 7f02212bb700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0fbe90 0x7f021c0fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:31.278 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.274+0000 7f02212bb700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0fbe90 0x7f021c0fc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39650/0 (socket says 172.21.15.58:39650) 2021-11-20T05:29:31.278 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.274+0000 7f02212bb700 1 -- 172.21.15.58:0/2168661625 learned_addr learned my addr 172.21.15.58:0/2168661625 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:31.279 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.274+0000 7f02212bb700 1 -- 172.21.15.58:0/2168661625 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c0fc8e0 con 0x7f021c0fbe90 2021-11-20T05:29:31.279 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.274+0000 7f02212bb700 1 --2- 172.21.15.58:0/2168661625 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0fbe90 0x7f021c0fc2b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f020c00b5f0 tx=0x7f020c006ee0).ready entity=mon.0 client_cookie=f24f09557ca33de6 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:31.279 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.274+0000 7f020bfff700 1 -- 172.21.15.58:0/2168661625 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f020c00d600 con 0x7f021c0fbe90 2021-11-20T05:29:31.279 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.274+0000 7f020bfff700 1 -- 172.21.15.58:0/2168661625 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f020c00dbd0 con 0x7f021c0fbe90 2021-11-20T05:29:31.280 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.274+0000 7f020bfff700 1 -- 172.21.15.58:0/2168661625 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f020c011c00 con 0x7f021c0fbe90 2021-11-20T05:29:31.280 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.275+0000 7f02222bd700 1 -- 172.21.15.58:0/2168661625 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0fbe90 msgr2=0x7f021c0fc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:31.280 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.275+0000 7f02222bd700 1 --2- 172.21.15.58:0/2168661625 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0fbe90 0x7f021c0fc2b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f020c00b5f0 tx=0x7f020c006ee0).stop 2021-11-20T05:29:31.280 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.275+0000 7f02222bd700 1 -- 172.21.15.58:0/2168661625 shutdown_connections 2021-11-20T05:29:31.281 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.275+0000 7f02222bd700 1 --2- 172.21.15.58:0/2168661625 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0fbe90 0x7f021c0fc2b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:31.281 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.275+0000 7f02222bd700 1 -- 172.21.15.58:0/2168661625 >> 172.21.15.58:0/2168661625 conn(0x7f021c0f5870 msgr2=0x7f021c0f7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:31.281 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.275+0000 7f02222bd700 1 -- 172.21.15.58:0/2168661625 shutdown_connections 2021-11-20T05:29:31.281 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.275+0000 7f02222bd700 1 -- 172.21.15.58:0/2168661625 wait complete. 2021-11-20T05:29:31.282 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.276+0000 7f02222bd700 1 Processor -- start 2021-11-20T05:29:31.282 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.276+0000 7f02222bd700 1 -- start start 2021-11-20T05:29:31.282 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.276+0000 7f02222bd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0fbe90 0x7f021c10a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:31.282 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.276+0000 7f02222bd700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c10ad70 con 0x7f021c0fbe90 2021-11-20T05:29:31.282 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.276+0000 7f02212bb700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0fbe90 0x7f021c10a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:31.283 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.276+0000 7f02212bb700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0fbe90 0x7f021c10a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39652/0 (socket says 172.21.15.58:39652) 2021-11-20T05:29:31.283 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.276+0000 7f02212bb700 1 -- 172.21.15.58:0/813721673 learned_addr learned my addr 172.21.15.58:0/813721673 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:31.283 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.277+0000 7f02212bb700 1 -- 172.21.15.58:0/813721673 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020c00b200 con 0x7f021c0fbe90 2021-11-20T05:29:31.283 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.277+0000 7f02212bb700 1 --2- 172.21.15.58:0/813721673 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0fbe90 0x7f021c10a880 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f020c000c00 tx=0x7f020c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:31.284 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.277+0000 7f0209ffb700 1 -- 172.21.15.58:0/813721673 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f020c011690 con 0x7f021c0fbe90 2021-11-20T05:29:31.284 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.277+0000 7f0209ffb700 1 -- 172.21.15.58:0/813721673 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f020c00d9f0 con 0x7f021c0fbe90 2021-11-20T05:29:31.284 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.277+0000 7f02222bd700 1 -- 172.21.15.58:0/813721673 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021c10af70 con 0x7f021c0fbe90 2021-11-20T05:29:31.284 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.277+0000 7f0209ffb700 1 -- 172.21.15.58:0/813721673 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f020c01b710 con 0x7f021c0fbe90 2021-11-20T05:29:31.284 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.277+0000 7f02222bd700 1 -- 172.21.15.58:0/813721673 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f021c10b380 con 0x7f021c0fbe90 2021-11-20T05:29:31.286 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.278+0000 7f0209ffb700 1 -- 172.21.15.58:0/813721673 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f020c00f070 con 0x7f021c0fbe90 2021-11-20T05:29:31.286 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.278+0000 7f0209ffb700 1 --2- 172.21.15.58:0/813721673 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f020402fd50 0x7f0204032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:31.287 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.278+0000 7f0209ffb700 1 -- 172.21.15.58:0/813721673 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f020c041170 con 0x7f021c0fbe90 2021-11-20T05:29:31.287 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.279+0000 7f02222bd700 1 -- 172.21.15.58:0/813721673 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f021c045ad0 con 0x7f021c0fbe90 2021-11-20T05:29:31.287 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.279+0000 7f0220aba700 1 --2- 172.21.15.58:0/813721673 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f020402fd50 0x7f0204032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:31.287 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.279+0000 7f0220aba700 1 --2- 172.21.15.58:0/813721673 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f020402fd50 0x7f0204032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f02180096a0 tx=0x7f0218006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:31.288 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.283+0000 7f0209ffb700 1 -- 172.21.15.58:0/813721673 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f020c016070 con 0x7f021c0fbe90 2021-11-20T05:29:31.630 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.630+0000 7f02222bd700 1 -- 172.21.15.58:0/813721673 --> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f021c104760 con 0x7f020402fd50 2021-11-20T05:29:31.634 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.634+0000 7f0209ffb700 1 -- 172.21.15.58:0/813721673 <== mgr.14116 v2:172.21.15.58:6800/1261811851 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f021c104760 con 0x7f020402fd50 2021-11-20T05:29:31.634 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: Scheduled mon update... 2021-11-20T05:29:31.637 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.637+0000 7f02222bd700 1 -- 172.21.15.58:0/813721673 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f020402fd50 msgr2=0x7f0204032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:31.638 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.637+0000 7f02222bd700 1 --2- 172.21.15.58:0/813721673 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f020402fd50 0x7f0204032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f02180096a0 tx=0x7f0218006b40).stop 2021-11-20T05:29:31.638 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.637+0000 7f02222bd700 1 -- 172.21.15.58:0/813721673 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0fbe90 msgr2=0x7f021c10a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:31.638 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.637+0000 7f02222bd700 1 --2- 172.21.15.58:0/813721673 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0fbe90 0x7f021c10a880 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f020c000c00 tx=0x7f020c006ee0).stop 2021-11-20T05:29:31.638 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.637+0000 7f02222bd700 1 -- 172.21.15.58:0/813721673 shutdown_connections 2021-11-20T05:29:31.639 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.637+0000 7f02222bd700 1 --2- 172.21.15.58:0/813721673 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f020402fd50 0x7f0204032210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:31.639 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.637+0000 7f02222bd700 1 --2- 172.21.15.58:0/813721673 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0fbe90 0x7f021c10a880 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:31.639 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.637+0000 7f02222bd700 1 -- 172.21.15.58:0/813721673 >> 172.21.15.58:0/813721673 conn(0x7f021c0f5870 msgr2=0x7f021c0f7430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:31.639 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.637+0000 7f02222bd700 1 -- 172.21.15.58:0/813721673 shutdown_connections 2021-11-20T05:29:31.640 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:31.637+0000 7f02222bd700 1 -- 172.21.15.58:0/813721673 wait complete. 2021-11-20T05:29:32.219 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:32 smithi058 conmon[33537]: cephadm 2021-11-20T05:29:30.382250+0000 mgr.smithi058.zagdlk (mgr.14116) 9 : cephadm [INF] Added host smithi058 2021-11-20T05:29:32.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:32 smithi058 conmon[33537]: audit 2021-11-20T05:29:31.635263+0000 mon.smithi058 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:32.221 INFO:teuthology.orchestra.run.smithi058.stderr:Deploying mgr service with default placement... 2021-11-20T05:29:32.221 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2021-11-20T05:29:32.946 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.943+0000 7fa116976700 1 Processor -- start 2021-11-20T05:29:32.947 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.943+0000 7fa116976700 1 -- start start 2021-11-20T05:29:32.947 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.943+0000 7fa116976700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa11006fcd0 0x7fa1100fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:32.947 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.943+0000 7fa116976700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1100fed20 con 0x7fa11006fcd0 2021-11-20T05:29:32.947 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.944+0000 7fa10ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa11006fcd0 0x7fa1100fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:32.948 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.944+0000 7fa10ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa11006fcd0 0x7fa1100fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39656/0 (socket says 172.21.15.58:39656) 2021-11-20T05:29:32.948 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.944+0000 7fa10ffff700 1 -- 172.21.15.58:0/768579729 learned_addr learned my addr 172.21.15.58:0/768579729 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:32.948 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.944+0000 7fa10ffff700 1 -- 172.21.15.58:0/768579729 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1100fee60 con 0x7fa11006fcd0 2021-11-20T05:29:32.948 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.944+0000 7fa10ffff700 1 --2- 172.21.15.58:0/768579729 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa11006fcd0 0x7fa1100fe7a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fa10000fa00 tx=0x7fa100006ee0).ready entity=mon.0 client_cookie=55e1dc5fe7103271 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:32.948 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.944+0000 7fa10effd700 1 -- 172.21.15.58:0/768579729 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa10000f3a0 con 0x7fa11006fcd0 2021-11-20T05:29:32.949 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.945+0000 7fa10effd700 1 -- 172.21.15.58:0/768579729 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa100009ae0 con 0x7fa11006fcd0 2021-11-20T05:29:32.949 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.945+0000 7fa10effd700 1 -- 172.21.15.58:0/768579729 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa10000ebc0 con 0x7fa11006fcd0 2021-11-20T05:29:32.949 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.945+0000 7fa116976700 1 -- 172.21.15.58:0/768579729 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa11006fcd0 msgr2=0x7fa1100fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:32.950 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.945+0000 7fa116976700 1 --2- 172.21.15.58:0/768579729 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa11006fcd0 0x7fa1100fe7a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fa10000fa00 tx=0x7fa100006ee0).stop 2021-11-20T05:29:32.950 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.945+0000 7fa116976700 1 -- 172.21.15.58:0/768579729 shutdown_connections 2021-11-20T05:29:32.950 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.945+0000 7fa116976700 1 --2- 172.21.15.58:0/768579729 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa11006fcd0 0x7fa1100fe7a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:32.950 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.945+0000 7fa116976700 1 -- 172.21.15.58:0/768579729 >> 172.21.15.58:0/768579729 conn(0x7fa1100f6280 msgr2=0x7fa1100f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:32.951 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.946+0000 7fa116976700 1 -- 172.21.15.58:0/768579729 shutdown_connections 2021-11-20T05:29:32.951 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.946+0000 7fa116976700 1 -- 172.21.15.58:0/768579729 wait complete. 2021-11-20T05:29:32.951 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.946+0000 7fa116976700 1 Processor -- start 2021-11-20T05:29:32.951 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.946+0000 7fa116976700 1 -- start start 2021-11-20T05:29:32.953 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.947+0000 7fa116976700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa11006fcd0 0x7fa1101087a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:32.953 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.947+0000 7fa116976700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa110108c90 con 0x7fa11006fcd0 2021-11-20T05:29:32.954 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.947+0000 7fa10ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa11006fcd0 0x7fa1101087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:32.955 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.947+0000 7fa10ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa11006fcd0 0x7fa1101087a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39658/0 (socket says 172.21.15.58:39658) 2021-11-20T05:29:32.955 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.947+0000 7fa10ffff700 1 -- 172.21.15.58:0/2593222019 learned_addr learned my addr 172.21.15.58:0/2593222019 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:32.955 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.947+0000 7fa10ffff700 1 -- 172.21.15.58:0/2593222019 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10000f610 con 0x7fa11006fcd0 2021-11-20T05:29:32.956 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.947+0000 7fa10ffff700 1 --2- 172.21.15.58:0/2593222019 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa11006fcd0 0x7fa1101087a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa100006d40 tx=0x7fa100006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:32.956 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.948+0000 7fa10cff9700 1 -- 172.21.15.58:0/2593222019 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa100021440 con 0x7fa11006fcd0 2021-11-20T05:29:32.956 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.948+0000 7fa116976700 1 -- 172.21.15.58:0/2593222019 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa110108e90 con 0x7fa11006fcd0 2021-11-20T05:29:32.956 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.948+0000 7fa10cff9700 1 -- 172.21.15.58:0/2593222019 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa100021a50 con 0x7fa11006fcd0 2021-11-20T05:29:32.957 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.948+0000 7fa10cff9700 1 -- 172.21.15.58:0/2593222019 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1000188c0 con 0x7fa11006fcd0 2021-11-20T05:29:32.957 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.948+0000 7fa116976700 1 -- 172.21.15.58:0/2593222019 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1101092a0 con 0x7fa11006fcd0 2021-11-20T05:29:32.957 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.949+0000 7fa116976700 1 -- 172.21.15.58:0/2593222019 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa110045ad0 con 0x7fa11006fcd0 2021-11-20T05:29:32.957 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.949+0000 7fa10cff9700 1 -- 172.21.15.58:0/2593222019 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa100018a20 con 0x7fa11006fcd0 2021-11-20T05:29:32.958 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.949+0000 7fa10cff9700 1 --2- 172.21.15.58:0/2593222019 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fa0fc02fd50 0x7fa0fc032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:32.958 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.949+0000 7fa10cff9700 1 -- 172.21.15.58:0/2593222019 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa100044f50 con 0x7fa11006fcd0 2021-11-20T05:29:32.958 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.949+0000 7fa10f7fe700 1 --2- 172.21.15.58:0/2593222019 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fa0fc02fd50 0x7fa0fc032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:32.958 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.950+0000 7fa10f7fe700 1 --2- 172.21.15.58:0/2593222019 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fa0fc02fd50 0x7fa0fc032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fa1040096a0 tx=0x7fa104006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:32.959 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:32.954+0000 7fa10cff9700 1 -- 172.21.15.58:0/2593222019 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa100018cb0 con 0x7fa11006fcd0 2021-11-20T05:29:33.219 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:33 smithi058 conmon[33537]: audit 2021-11-20T05:29:31.632024+0000 mgr.smithi058.zagdlk (mgr.14116) 10 : audit 2021-11-20T05:29:33.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:33 smithi058 conmon[33537]: [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:29:33.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:33 smithi058 conmon[33537]: cephadm 2021-11-20T05:29:31.632906+0000 mgr.smithi058.zagdlk (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2021-11-20T05:29:33.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:33 smithi058 conmon[33537]: audit 2021-11-20T05:29:32.778167+0000 mon.smithi058 (mon.0) 58 : 2021-11-20T05:29:33.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:33 smithi058 conmon[33537]: audit [INF] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:33.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:33 smithi058 conmon[33537]: audit 2021-11-20T05:29: 2021-11-20T05:29:33.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:33 smithi058 conmon[33537]: 33.124065+0000 mon.smithi058 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:33.304 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:33.302+0000 7fa116976700 1 -- 172.21.15.58:0/2593222019 --> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fa110109be0 con 0x7fa0fc02fd50 2021-11-20T05:29:33.306 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:33.306+0000 7fa10cff9700 1 -- 172.21.15.58:0/2593222019 <== mgr.14116 v2:172.21.15.58:6800/1261811851 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa110109be0 con 0x7fa0fc02fd50 2021-11-20T05:29:33.306 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: Scheduled mgr update... 2021-11-20T05:29:33.309 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:33.308+0000 7fa116976700 1 -- 172.21.15.58:0/2593222019 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fa0fc02fd50 msgr2=0x7fa0fc032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:33.309 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:33.308+0000 7fa116976700 1 --2- 172.21.15.58:0/2593222019 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fa0fc02fd50 0x7fa0fc032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fa1040096a0 tx=0x7fa104006b40).stop 2021-11-20T05:29:33.309 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:33.308+0000 7fa116976700 1 -- 172.21.15.58:0/2593222019 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa11006fcd0 msgr2=0x7fa1101087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:33.310 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:33.308+0000 7fa116976700 1 --2- 172.21.15.58:0/2593222019 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa11006fcd0 0x7fa1101087a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa100006d40 tx=0x7fa100006ee0).stop 2021-11-20T05:29:33.310 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:33.308+0000 7fa116976700 1 -- 172.21.15.58:0/2593222019 shutdown_connections 2021-11-20T05:29:33.310 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:33.308+0000 7fa116976700 1 --2- 172.21.15.58:0/2593222019 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7fa0fc02fd50 0x7fa0fc032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:33.310 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:33.308+0000 7fa116976700 1 --2- 172.21.15.58:0/2593222019 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa11006fcd0 0x7fa1101087a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:33.311 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:33.308+0000 7fa116976700 1 -- 172.21.15.58:0/2593222019 >> 172.21.15.58:0/2593222019 conn(0x7fa1100f6280 msgr2=0x7fa1100fcce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:33.311 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:33.309+0000 7fa116976700 1 -- 172.21.15.58:0/2593222019 shutdown_connections 2021-11-20T05:29:33.311 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:33.309+0000 7fa116976700 1 -- 172.21.15.58:0/2593222019 wait complete. 2021-11-20T05:29:33.759 INFO:teuthology.orchestra.run.smithi058.stderr:Deploying crash service with default placement... 2021-11-20T05:29:33.759 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2021-11-20T05:29:34.301 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:34 smithi058 conmon[33537]: audit 2021-11-20T05:29:33.303696+0000 mgr.smithi058.zagdlk (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:29:34.302 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:34 smithi058 conmon[33537]: cephadm 2021-11-20T05:29:33.304298+0000 mgr.smithi058.zagdlk (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2021-11-20T05:29:34.302 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:34 smithi058 conmon[33537]: audit 2021-11-20T05:29:33.306454+0000 mon.smithi058 (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:34.306 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.302+0000 7f777bca3700 1 Processor -- start 2021-11-20T05:29:34.306 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.302+0000 7f777bca3700 1 -- start start 2021-11-20T05:29:34.306 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.302+0000 7f777bca3700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7774071c90 0x7f7774070230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:34.306 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.302+0000 7f777bca3700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77740720b0 con 0x7f7774071c90 2021-11-20T05:29:34.307 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.302+0000 7f7779a3f700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7774071c90 0x7f7774070230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:34.307 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.302+0000 7f7779a3f700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7774071c90 0x7f7774070230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39662/0 (socket says 172.21.15.58:39662) 2021-11-20T05:29:34.307 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.302+0000 7f7779a3f700 1 -- 172.21.15.58:0/1110556348 learned_addr learned my addr 172.21.15.58:0/1110556348 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:34.307 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.303+0000 7f7779a3f700 1 -- 172.21.15.58:0/1110556348 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7774070720 con 0x7f7774071c90 2021-11-20T05:29:34.307 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.303+0000 7f7779a3f700 1 --2- 172.21.15.58:0/1110556348 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7774071c90 0x7f7774070230 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f777000b5f0 tx=0x7f7770006ee0).ready entity=mon.0 client_cookie=852674392a9ae33 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:34.308 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.303+0000 7f7778a3d700 1 -- 172.21.15.58:0/1110556348 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f777000d600 con 0x7f7774071c90 2021-11-20T05:29:34.308 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.303+0000 7f7778a3d700 1 -- 172.21.15.58:0/1110556348 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f777000dc10 con 0x7f7774071c90 2021-11-20T05:29:34.308 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.303+0000 7f7778a3d700 1 -- 172.21.15.58:0/1110556348 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7770011cd0 con 0x7f7774071c90 2021-11-20T05:29:34.308 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.304+0000 7f777bca3700 1 -- 172.21.15.58:0/1110556348 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7774071c90 msgr2=0x7f7774070230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:34.309 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.304+0000 7f777bca3700 1 --2- 172.21.15.58:0/1110556348 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7774071c90 0x7f7774070230 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f777000b5f0 tx=0x7f7770006ee0).stop 2021-11-20T05:29:34.309 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.304+0000 7f777bca3700 1 -- 172.21.15.58:0/1110556348 shutdown_connections 2021-11-20T05:29:34.309 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.304+0000 7f777bca3700 1 --2- 172.21.15.58:0/1110556348 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7774071c90 0x7f7774070230 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:34.309 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.304+0000 7f777bca3700 1 -- 172.21.15.58:0/1110556348 >> 172.21.15.58:0/1110556348 conn(0x7f77740f73e0 msgr2=0x7f77740f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:34.309 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.304+0000 7f777bca3700 1 -- 172.21.15.58:0/1110556348 shutdown_connections 2021-11-20T05:29:34.310 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.304+0000 7f777bca3700 1 -- 172.21.15.58:0/1110556348 wait complete. 2021-11-20T05:29:34.310 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.305+0000 7f777bca3700 1 Processor -- start 2021-11-20T05:29:34.310 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.305+0000 7f777bca3700 1 -- start start 2021-11-20T05:29:34.310 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.305+0000 7f777bca3700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7774071c90 0x7f777410cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:34.311 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.305+0000 7f777bca3700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777410d0e0 con 0x7f7774071c90 2021-11-20T05:29:34.311 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.305+0000 7f7779a3f700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7774071c90 0x7f777410cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:34.311 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.305+0000 7f7779a3f700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7774071c90 0x7f777410cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39664/0 (socket says 172.21.15.58:39664) 2021-11-20T05:29:34.311 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.305+0000 7f7779a3f700 1 -- 172.21.15.58:0/1267501176 learned_addr learned my addr 172.21.15.58:0/1267501176 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:34.311 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.305+0000 7f7779a3f700 1 -- 172.21.15.58:0/1267501176 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777000b200 con 0x7f7774071c90 2021-11-20T05:29:34.312 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.305+0000 7f7779a3f700 1 --2- 172.21.15.58:0/1267501176 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7774071c90 0x7f777410cbf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f7770006800 tx=0x7f7770006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:34.312 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.306+0000 7f77627fc700 1 -- 172.21.15.58:0/1267501176 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7770011c90 con 0x7f7774071c90 2021-11-20T05:29:34.312 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.306+0000 7f77627fc700 1 -- 172.21.15.58:0/1267501176 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f777001b8c0 con 0x7f7774071c90 2021-11-20T05:29:34.312 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.306+0000 7f777bca3700 1 -- 172.21.15.58:0/1267501176 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f777410d2e0 con 0x7f7774071c90 2021-11-20T05:29:34.312 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.306+0000 7f77627fc700 1 -- 172.21.15.58:0/1267501176 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f777001a880 con 0x7f7774071c90 2021-11-20T05:29:34.313 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.306+0000 7f777bca3700 1 -- 172.21.15.58:0/1267501176 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f777410d6f0 con 0x7f7774071c90 2021-11-20T05:29:34.314 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.307+0000 7f77627fc700 1 -- 172.21.15.58:0/1267501176 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f777000f070 con 0x7f7774071c90 2021-11-20T05:29:34.314 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.307+0000 7f77627fc700 1 --2- 172.21.15.58:0/1267501176 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f775c02fdb0 0x7f775c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:34.314 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.307+0000 7f77627fc700 1 -- 172.21.15.58:0/1267501176 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f7770040960 con 0x7f7774071c90 2021-11-20T05:29:34.315 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.307+0000 7f777bca3700 1 -- 172.21.15.58:0/1267501176 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7774045ad0 con 0x7f7774071c90 2021-11-20T05:29:34.315 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.307+0000 7f777923e700 1 --2- 172.21.15.58:0/1267501176 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f775c02fdb0 0x7f775c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:34.315 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.307+0000 7f777923e700 1 --2- 172.21.15.58:0/1267501176 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f775c02fdb0 0x7f775c032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f77640096a0 tx=0x7f7764006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:34.315 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.311+0000 7f77627fc700 1 -- 172.21.15.58:0/1267501176 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7770016070 con 0x7f7774071c90 2021-11-20T05:29:34.652 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.652+0000 7f777bca3700 1 -- 172.21.15.58:0/1267501176 --> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f777410dfd0 con 0x7f775c02fdb0 2021-11-20T05:29:34.657 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.656+0000 7f77627fc700 1 -- 172.21.15.58:0/1267501176 <== mgr.14116 v2:172.21.15.58:6800/1261811851 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f777410dfd0 con 0x7f775c02fdb0 2021-11-20T05:29:34.657 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: Scheduled crash update... 2021-11-20T05:29:34.660 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.658+0000 7f777bca3700 1 -- 172.21.15.58:0/1267501176 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f775c02fdb0 msgr2=0x7f775c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:34.660 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.658+0000 7f777bca3700 1 --2- 172.21.15.58:0/1267501176 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f775c02fdb0 0x7f775c032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f77640096a0 tx=0x7f7764006b40).stop 2021-11-20T05:29:34.661 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.658+0000 7f777bca3700 1 -- 172.21.15.58:0/1267501176 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7774071c90 msgr2=0x7f777410cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:34.661 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.658+0000 7f777bca3700 1 --2- 172.21.15.58:0/1267501176 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7774071c90 0x7f777410cbf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f7770006800 tx=0x7f7770006ee0).stop 2021-11-20T05:29:34.662 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.658+0000 7f777bca3700 1 -- 172.21.15.58:0/1267501176 shutdown_connections 2021-11-20T05:29:34.662 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.658+0000 7f777bca3700 1 --2- 172.21.15.58:0/1267501176 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f775c02fdb0 0x7f775c032270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:34.662 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.658+0000 7f777bca3700 1 --2- 172.21.15.58:0/1267501176 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7774071c90 0x7f777410cbf0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:34.663 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.658+0000 7f777bca3700 1 -- 172.21.15.58:0/1267501176 >> 172.21.15.58:0/1267501176 conn(0x7f77740f73e0 msgr2=0x7f77740f8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:34.663 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.659+0000 7f777bca3700 1 -- 172.21.15.58:0/1267501176 shutdown_connections 2021-11-20T05:29:34.664 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:34.659+0000 7f777bca3700 1 -- 172.21.15.58:0/1267501176 wait complete. 2021-11-20T05:29:34.993 INFO:teuthology.orchestra.run.smithi058.stderr:Enabling mgr prometheus module... 2021-11-20T05:29:34.994 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2021-11-20T05:29:35.501 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.500+0000 7f0b1be7f700 1 Processor -- start 2021-11-20T05:29:35.501 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.500+0000 7f0b1be7f700 1 -- start start 2021-11-20T05:29:35.502 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.501+0000 7f0b1be7f700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1406fc90 0x7f0b140fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:35.502 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.501+0000 7f0b1be7f700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b140fed60 con 0x7f0b1406fc90 2021-11-20T05:29:35.502 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.501+0000 7f0b19c1b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1406fc90 0x7f0b140fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:35.502 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.501+0000 7f0b19c1b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1406fc90 0x7f0b140fe7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39668/0 (socket says 172.21.15.58:39668) 2021-11-20T05:29:35.502 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.501+0000 7f0b19c1b700 1 -- 172.21.15.58:0/2420537137 learned_addr learned my addr 172.21.15.58:0/2420537137 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:35.503 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.502+0000 7f0b19c1b700 1 -- 172.21.15.58:0/2420537137 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b140feea0 con 0x7f0b1406fc90 2021-11-20T05:29:35.504 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.502+0000 7f0b19c1b700 1 --2- 172.21.15.58:0/2420537137 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1406fc90 0x7f0b140fe7e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f0b04009960 tx=0x7f0b040092d0).ready entity=mon.0 client_cookie=527205c42dea51d2 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:35.505 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.502+0000 7f0b18c19700 1 -- 172.21.15.58:0/2420537137 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b0400b9c0 con 0x7f0b1406fc90 2021-11-20T05:29:35.505 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.502+0000 7f0b18c19700 1 -- 172.21.15.58:0/2420537137 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b04011400 con 0x7f0b1406fc90 2021-11-20T05:29:35.505 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.502+0000 7f0b18c19700 1 -- 172.21.15.58:0/2420537137 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b04010580 con 0x7f0b1406fc90 2021-11-20T05:29:35.505 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.503+0000 7f0b1be7f700 1 -- 172.21.15.58:0/2420537137 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1406fc90 msgr2=0x7f0b140fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:35.505 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.503+0000 7f0b1be7f700 1 --2- 172.21.15.58:0/2420537137 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1406fc90 0x7f0b140fe7e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f0b04009960 tx=0x7f0b040092d0).stop 2021-11-20T05:29:35.506 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.503+0000 7f0b1be7f700 1 -- 172.21.15.58:0/2420537137 shutdown_connections 2021-11-20T05:29:35.506 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.503+0000 7f0b1be7f700 1 --2- 172.21.15.58:0/2420537137 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1406fc90 0x7f0b140fe7e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:35.506 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.503+0000 7f0b1be7f700 1 -- 172.21.15.58:0/2420537137 >> 172.21.15.58:0/2420537137 conn(0x7f0b140f6280 msgr2=0x7f0b140f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:35.506 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.504+0000 7f0b1be7f700 1 -- 172.21.15.58:0/2420537137 shutdown_connections 2021-11-20T05:29:35.507 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.504+0000 7f0b1be7f700 1 -- 172.21.15.58:0/2420537137 wait complete. 2021-11-20T05:29:35.508 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.504+0000 7f0b1be7f700 1 Processor -- start 2021-11-20T05:29:35.509 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.505+0000 7f0b1be7f700 1 -- start start 2021-11-20T05:29:35.509 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.505+0000 7f0b1be7f700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1406fc90 0x7f0b140fc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:35.509 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.505+0000 7f0b1be7f700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b140fa6e0 con 0x7f0b1406fc90 2021-11-20T05:29:35.509 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.505+0000 7f0b19c1b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1406fc90 0x7f0b140fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:35.509 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.505+0000 7f0b19c1b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1406fc90 0x7f0b140fc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39670/0 (socket says 172.21.15.58:39670) 2021-11-20T05:29:35.510 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.505+0000 7f0b19c1b700 1 -- 172.21.15.58:0/3719173380 learned_addr learned my addr 172.21.15.58:0/3719173380 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:35.510 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.505+0000 7f0b19c1b700 1 -- 172.21.15.58:0/3719173380 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b04009570 con 0x7f0b1406fc90 2021-11-20T05:29:35.510 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.505+0000 7f0b19c1b700 1 --2- 172.21.15.58:0/3719173380 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1406fc90 0x7f0b140fc260 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f0b04006dc0 tx=0x7f0b040092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:35.510 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.506+0000 7f0b127fc700 1 -- 172.21.15.58:0/3719173380 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b040103a0 con 0x7f0b1406fc90 2021-11-20T05:29:35.511 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.506+0000 7f0b127fc700 1 -- 172.21.15.58:0/3719173380 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b04010cf0 con 0x7f0b1406fc90 2021-11-20T05:29:35.511 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.506+0000 7f0b1be7f700 1 -- 172.21.15.58:0/3719173380 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b140fa880 con 0x7f0b1406fc90 2021-11-20T05:29:35.511 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.506+0000 7f0b1be7f700 1 -- 172.21.15.58:0/3719173380 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b140fac90 con 0x7f0b1406fc90 2021-11-20T05:29:35.511 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.506+0000 7f0b127fc700 1 -- 172.21.15.58:0/3719173380 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b0401baf0 con 0x7f0b1406fc90 2021-11-20T05:29:35.511 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.507+0000 7f0b127fc700 1 -- 172.21.15.58:0/3719173380 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f0b0400f070 con 0x7f0b1406fc90 2021-11-20T05:29:35.512 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.507+0000 7f0b127fc700 1 --2- 172.21.15.58:0/3719173380 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f0b0002fdb0 0x7f0b00032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:35.512 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.507+0000 7f0b127fc700 1 -- 172.21.15.58:0/3719173380 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0b04042750 con 0x7f0b1406fc90 2021-11-20T05:29:35.512 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.507+0000 7f0b1be7f700 1 -- 172.21.15.58:0/3719173380 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0af4004fa0 con 0x7f0b1406fc90 2021-11-20T05:29:35.512 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.507+0000 7f0b1941a700 1 --2- 172.21.15.58:0/3719173380 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f0b0002fdb0 0x7f0b00032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:35.513 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.508+0000 7f0b1941a700 1 --2- 172.21.15.58:0/3719173380 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f0b0002fdb0 0x7f0b00032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f0b080096a0 tx=0x7f0b08006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:35.513 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.512+0000 7f0b127fc700 1 -- 172.21.15.58:0/3719173380 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b04024cb0 con 0x7f0b1406fc90 2021-11-20T05:29:35.876 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:35 smithi058 conmon[33537]: audit 2021-11-20T05:29:34.654438+0000 mgr.smithi058.zagdlk (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:29:35.876 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:35 smithi058 conmon[33537]: cephadm 2021-11-20T05:29:34.655169+0000 mgr.smithi058.zagdlk (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2021-11-20T05:29:35.877 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:35 smithi058 conmon[33537]: audit 2021-11-20T05:29:34.657286+0000 mon.smithi058 (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:35.877 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:35 smithi058 conmon[33537]: audit 2021-11-20T05:29:34.705578+0000 mon.smithi058 (mon.0) 62 : audit [INF] from='mgr.14116 172.21.15.58:0/3239928023' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:35.877 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:35.876+0000 7f0b1be7f700 1 -- 172.21.15.58:0/3719173380 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f0af4005670 con 0x7f0b1406fc90 2021-11-20T05:29:36.660 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:36.660+0000 7f0b127fc700 1 -- 172.21.15.58:0/3719173380 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f0b0401bc50 con 0x7f0b1406fc90 2021-11-20T05:29:36.661 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:36.661+0000 7f0b127fc700 1 -- 172.21.15.58:0/3719173380 <== mon.0 v2:172.21.15.58:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f0b04024cb0 con 0x7f0b1406fc90 2021-11-20T05:29:36.664 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:36.664+0000 7f0b1be7f700 1 -- 172.21.15.58:0/3719173380 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f0b0002fdb0 msgr2=0x7f0b00032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:36.664 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:36.664+0000 7f0b1be7f700 1 --2- 172.21.15.58:0/3719173380 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f0b0002fdb0 0x7f0b00032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f0b080096a0 tx=0x7f0b08006b40).stop 2021-11-20T05:29:36.664 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:36.664+0000 7f0b1be7f700 1 -- 172.21.15.58:0/3719173380 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1406fc90 msgr2=0x7f0b140fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:36.664 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:36.664+0000 7f0b1be7f700 1 --2- 172.21.15.58:0/3719173380 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1406fc90 0x7f0b140fc260 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f0b04006dc0 tx=0x7f0b040092d0).stop 2021-11-20T05:29:36.665 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:36.664+0000 7f0b1be7f700 1 -- 172.21.15.58:0/3719173380 shutdown_connections 2021-11-20T05:29:36.666 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:36.664+0000 7f0b1be7f700 1 --2- 172.21.15.58:0/3719173380 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7f0b0002fdb0 0x7f0b00032270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:36.666 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:36.664+0000 7f0b1be7f700 1 --2- 172.21.15.58:0/3719173380 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1406fc90 0x7f0b140fc260 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:36.666 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:36.664+0000 7f0b1be7f700 1 -- 172.21.15.58:0/3719173380 >> 172.21.15.58:0/3719173380 conn(0x7f0b140f6280 msgr2=0x7f0b140f7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:36.666 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:36.665+0000 7f0b1be7f700 1 -- 172.21.15.58:0/3719173380 shutdown_connections 2021-11-20T05:29:36.667 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:36.665+0000 7f0b1be7f700 1 -- 172.21.15.58:0/3719173380 wait complete. 2021-11-20T05:29:36.969 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:36 smithi058 conmon[33537]: audit 2021-11-20T05:29:35.877838+0000 mon.smithi058 (mon. 2021-11-20T05:29:36.969 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:36 smithi058 conmon[33537]: 0) 63 : audit [INF] from='client.? 172.21.15.58:0/3719173380' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2021-11-20T05:29:37.033 INFO:teuthology.orchestra.run.smithi058.stderr:Deploying prometheus service with default placement... 2021-11-20T05:29:37.034 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2021-11-20T05:29:37.545 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.542+0000 7ff15b299700 1 Processor -- start 2021-11-20T05:29:37.545 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.543+0000 7ff15b299700 1 -- start start 2021-11-20T05:29:37.546 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.543+0000 7ff15b299700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff15406fcd0 0x7ff154100ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:37.546 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.543+0000 7ff15b299700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540fc620 con 0x7ff15406fcd0 2021-11-20T05:29:37.546 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.543+0000 7ff159035700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff15406fcd0 0x7ff154100ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:37.547 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.543+0000 7ff159035700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff15406fcd0 0x7ff154100ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39684/0 (socket says 172.21.15.58:39684) 2021-11-20T05:29:37.547 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.543+0000 7ff159035700 1 -- 172.21.15.58:0/1044964834 learned_addr learned my addr 172.21.15.58:0/1044964834 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:37.547 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.543+0000 7ff159035700 1 -- 172.21.15.58:0/1044964834 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff154101090 con 0x7ff15406fcd0 2021-11-20T05:29:37.547 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.543+0000 7ff159035700 1 --2- 172.21.15.58:0/1044964834 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff15406fcd0 0x7ff154100ba0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7ff15000b5f0 tx=0x7ff150006ee0).ready entity=mon.0 client_cookie=4bdbe3e8f33170c3 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:37.548 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.544+0000 7ff147fff700 1 -- 172.21.15.58:0/1044964834 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff15000d600 con 0x7ff15406fcd0 2021-11-20T05:29:37.548 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.544+0000 7ff147fff700 1 -- 172.21.15.58:0/1044964834 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff15000dc10 con 0x7ff15406fcd0 2021-11-20T05:29:37.548 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.544+0000 7ff147fff700 1 -- 172.21.15.58:0/1044964834 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff150011cd0 con 0x7ff15406fcd0 2021-11-20T05:29:37.548 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.544+0000 7ff15b299700 1 -- 172.21.15.58:0/1044964834 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff15406fcd0 msgr2=0x7ff154100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:37.548 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.544+0000 7ff15b299700 1 --2- 172.21.15.58:0/1044964834 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff15406fcd0 0x7ff154100ba0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7ff15000b5f0 tx=0x7ff150006ee0).stop 2021-11-20T05:29:37.549 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.545+0000 7ff15b299700 1 -- 172.21.15.58:0/1044964834 shutdown_connections 2021-11-20T05:29:37.549 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.545+0000 7ff15b299700 1 --2- 172.21.15.58:0/1044964834 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff15406fcd0 0x7ff154100ba0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:37.549 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.545+0000 7ff15b299700 1 -- 172.21.15.58:0/1044964834 >> 172.21.15.58:0/1044964834 conn(0x7ff1540f6280 msgr2=0x7ff1540f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:37.549 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.545+0000 7ff15b299700 1 -- 172.21.15.58:0/1044964834 shutdown_connections 2021-11-20T05:29:37.550 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.545+0000 7ff15b299700 1 -- 172.21.15.58:0/1044964834 wait complete. 2021-11-20T05:29:37.550 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.545+0000 7ff15b299700 1 Processor -- start 2021-11-20T05:29:37.550 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.545+0000 7ff15b299700 1 -- start start 2021-11-20T05:29:37.554 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.546+0000 7ff15b299700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff15406fcd0 0x7ff15410ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:37.554 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.546+0000 7ff15b299700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15410d1c0 con 0x7ff15406fcd0 2021-11-20T05:29:37.555 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.546+0000 7ff159035700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff15406fcd0 0x7ff15410ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:37.555 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.546+0000 7ff159035700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff15406fcd0 0x7ff15410ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39686/0 (socket says 172.21.15.58:39686) 2021-11-20T05:29:37.555 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.546+0000 7ff159035700 1 -- 172.21.15.58:0/642146589 learned_addr learned my addr 172.21.15.58:0/642146589 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:37.555 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.546+0000 7ff159035700 1 -- 172.21.15.58:0/642146589 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15000b200 con 0x7ff15406fcd0 2021-11-20T05:29:37.556 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.546+0000 7ff159035700 1 --2- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff15406fcd0 0x7ff15410ccd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ff15000bf40 tx=0x7ff150006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:37.556 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.546+0000 7ff145ffb700 1 -- 172.21.15.58:0/642146589 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff150011df0 con 0x7ff15406fcd0 2021-11-20T05:29:37.556 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.547+0000 7ff145ffb700 1 -- 172.21.15.58:0/642146589 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff15001b8c0 con 0x7ff15406fcd0 2021-11-20T05:29:37.556 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.547+0000 7ff15b299700 1 -- 172.21.15.58:0/642146589 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff15410d3c0 con 0x7ff15406fcd0 2021-11-20T05:29:37.557 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.547+0000 7ff145ffb700 1 -- 172.21.15.58:0/642146589 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff15001a8e0 con 0x7ff15406fcd0 2021-11-20T05:29:37.557 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.547+0000 7ff15b299700 1 -- 172.21.15.58:0/642146589 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15410d7d0 con 0x7ff15406fcd0 2021-11-20T05:29:37.557 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.548+0000 7ff145ffb700 1 -- 172.21.15.58:0/642146589 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7ff15000f070 con 0x7ff15406fcd0 2021-11-20T05:29:37.557 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.548+0000 7ff145ffb700 1 --2- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7ff13c02fdb0 0x7ff13c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:37.557 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.548+0000 7ff145ffb700 1 -- 172.21.15.58:0/642146589 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff150041e00 con 0x7ff15406fcd0 2021-11-20T05:29:37.558 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.548+0000 7ff158834700 1 -- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7ff13c02fdb0 msgr2=0x7ff13c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.58:6800/1261811851 2021-11-20T05:29:37.558 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.548+0000 7ff158834700 1 --2- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7ff13c02fdb0 0x7ff13c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T05:29:37.558 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.548+0000 7ff15b299700 1 -- 172.21.15.58:0/642146589 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff154045ad0 con 0x7ff15406fcd0 2021-11-20T05:29:37.559 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.556+0000 7ff145ffb700 1 -- 172.21.15.58:0/642146589 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff150024410 con 0x7ff15406fcd0 2021-11-20T05:29:37.748 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.749+0000 7ff158834700 1 -- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7ff13c02fdb0 msgr2=0x7ff13c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.58:6800/1261811851 2021-11-20T05:29:37.749 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.749+0000 7ff158834700 1 --2- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7ff13c02fdb0 0x7ff13c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T05:29:37.900 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:37 smithi058 conmon[33537]: audit 2021-11-20T05:29:36.662461+0000 mon.smithi058 ( 2021-11-20T05:29:37.901 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:37 smithi058 conmon[33537]: mon.0) 64 : audit [INF] from='client.? 172.21.15.58:0/3719173380' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2021-11-20T05:29:37.901 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:37 smithi058 conmon[33537]: cluster 2021-11-20T05:29:36.662532+0000 mon.smithi058 (mon.0) 65 : cluster [DBG] mgrmap e9: smithi058.zagdlk(active, since 14s) 2021-11-20T05:29:37.901 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:37.900+0000 7ff15b299700 1 -- 172.21.15.58:0/642146589 --> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7ff15410ed30 con 0x7ff13c02fdb0 2021-11-20T05:29:38.150 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:38.150+0000 7ff158834700 1 -- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7ff13c02fdb0 msgr2=0x7ff13c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.58:6800/1261811851 2021-11-20T05:29:38.150 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:38.150+0000 7ff158834700 1 --2- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7ff13c02fdb0 0x7ff13c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-20T05:29:38.951 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:38.951+0000 7ff158834700 1 -- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7ff13c02fdb0 msgr2=0x7ff13c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.58:6800/1261811851 2021-11-20T05:29:38.951 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:38.951+0000 7ff158834700 1 --2- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7ff13c02fdb0 0x7ff13c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-20T05:29:40.552 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:40.552+0000 7ff158834700 1 -- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7ff13c02fdb0 msgr2=0x7ff13c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.58:6800/1261811851 2021-11-20T05:29:40.552 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:40.552+0000 7ff158834700 1 --2- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7ff13c02fdb0 0x7ff13c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2021-11-20T05:29:42.684 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:42.683+0000 7ff145ffb700 1 -- 172.21.15.58:0/642146589 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7ff150024790 con 0x7ff15406fcd0 2021-11-20T05:29:42.685 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:42.683+0000 7ff145ffb700 1 -- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7ff13c02fdb0 msgr2=0x7ff13c032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T05:29:42.685 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:42.683+0000 7ff145ffb700 1 --2- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7ff13c02fdb0 0x7ff13c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:29:42.939 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:42 smithi058 conmon[33537]: cluster 2021-11-20T05:29:42.633663+0000 2021-11-20T05:29:42.940 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:42 smithi058 conmon[33537]: mon.smithi058 (mon.0) 66 : cluster [INF] Active manager daemon smithi058.zagdlk restarted 2021-11-20T05:29:42.940 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:42 smithi058 conmon[33537]: cluster 2021-11-20T05:29:42.634042+0000 mon.smithi058 (mon.0) 67 : cluster [INF] Activating manager daemon smithi058.zagdlk 2021-11-20T05:29:42.940 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:42 smithi058 conmon[33537]: cluster 2021-11-20T05:29:42 2021-11-20T05:29:42.941 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:42 smithi058 conmon[33537]: .635742+0000 mon.smithi058 (mon.0) 68 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2021-11-20T05:29:43.689 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:43.687+0000 7ff145ffb700 1 -- 172.21.15.58:0/642146589 <== mon.0 v2:172.21.15.58:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7ff150022af0 con 0x7ff15406fcd0 2021-11-20T05:29:43.690 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:43.687+0000 7ff145ffb700 1 --2- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7ff13c033c20 0x7ff13c036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:43.690 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:43.687+0000 7ff145ffb700 1 -- 172.21.15.58:0/642146589 --> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7ff15410ed30 con 0x7ff13c033c20 2021-11-20T05:29:43.690 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:43.688+0000 7ff158834700 1 --2- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7ff13c033c20 0x7ff13c036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:43.691 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:43.688+0000 7ff158834700 1 --2- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7ff13c033c20 0x7ff13c036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ff1480036d0 tx=0x7ff148006c60).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:43.694 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:43.693+0000 7ff145ffb700 1 -- 172.21.15.58:0/642146589 <== mgr.14144 v2:172.21.15.58:6800/2422532259 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7ff15410ed30 con 0x7ff13c033c20 2021-11-20T05:29:43.694 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: Scheduled prometheus update... 2021-11-20T05:29:43.697 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:43.696+0000 7ff15b299700 1 -- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7ff13c033c20 msgr2=0x7ff13c036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:43.697 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:43.696+0000 7ff15b299700 1 --2- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7ff13c033c20 0x7ff13c036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ff1480036d0 tx=0x7ff148006c60).stop 2021-11-20T05:29:43.698 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:43.696+0000 7ff15b299700 1 -- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff15406fcd0 msgr2=0x7ff15410ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:43.698 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:43.696+0000 7ff15b299700 1 --2- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff15406fcd0 0x7ff15410ccd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ff15000bf40 tx=0x7ff150006ee0).stop 2021-11-20T05:29:43.698 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:43.696+0000 7ff15b299700 1 -- 172.21.15.58:0/642146589 shutdown_connections 2021-11-20T05:29:43.698 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:43.696+0000 7ff15b299700 1 --2- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7ff13c033c20 0x7ff13c036010 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:43.699 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:43.696+0000 7ff15b299700 1 --2- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff15406fcd0 0x7ff15410ccd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:43.699 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:43.696+0000 7ff15b299700 1 --2- 172.21.15.58:0/642146589 >> [v2:172.21.15.58:6800/1261811851,v1:172.21.15.58:6801/1261811851] conn(0x7ff13c02fdb0 0x7ff13c032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:29:43.699 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:43.696+0000 7ff15b299700 1 -- 172.21.15.58:0/642146589 >> 172.21.15.58:0/642146589 conn(0x7ff1540f6280 msgr2=0x7ff1540f7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:43.699 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:43.697+0000 7ff15b299700 1 -- 172.21.15.58:0/642146589 shutdown_connections 2021-11-20T05:29:43.700 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:43.697+0000 7ff15b299700 1 -- 172.21.15.58:0/642146589 wait complete. 2021-11-20T05:29:43.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: cluster 2021-11-20T05:29:42 2021-11-20T05:29:43.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: .686282+0000 mon.smithi058 (mon.0) 69 : cluster [DBG] mgrmap e10: smithi058.zagdlk(active, starting, since 0.0523238s) 2021-11-20T05:29:43.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: audit 2021-11-20T05:29:42.689019+0000 mon.smithi058 (mon.0) 70 : audit [DBG] from='mgr.14144 172.21.15.58:0/3887114449' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi058"}]: dispatch 2021-11-20T05:29:43.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: 2021-11-20T05:29:43.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: audit 2021-11-20T 2021-11-20T05:29:43.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: 05:29:42.689741+0000 mon.smithi058 (mon.0) 71 2021-11-20T05:29:43.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: : audit [DBG] from='mgr.14144 172.21.15.58:0/3887114449' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mgr metadata", "who": "smithi058.zagdlk", "id": "smithi058.zagdlk"}]: dispatch 2021-11-20T05:29:43.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: audit 2021-11-20T05:29:43.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: 2021-11-20T05:29:42 2021-11-20T05:29:43.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: .690677+0000 mon.smithi058 (mon.0) 2021-11-20T05:29:43.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: 72 : audit [DBG] from='mgr.14144 172.21.15.58:0/3887114449' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-20T05:29:43.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:29:43.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: 11-20T05:29:42.690840+0000 mon.smithi058 2021-11-20T05:29:43.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.58:0/3887114449' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-20T05:29:43.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:29:43.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: 11-20T05:29:42.690991+0000 2021-11-20T05:29:43.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: mon.smithi058 (mon.0) 74 : audit [DBG] 2021-11-20T05:29:43.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: from='mgr.14144 172.21.15.58:0/3887114449' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-20T05:29:43.975 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: cluster 2021-11-20T05:29:43.975 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: 2021-11-20T05:29:42.693119+0000 2021-11-20T05:29:43.975 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: mon.smithi058 (mon.0) 75 : cluster [INF] Manager daemon smithi058.zagdlk is now available 2021-11-20T05:29:43.975 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: 2021-11-20T05:29:43.976 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: audit 2021-11-20T05:29: 2021-11-20T05:29:43.976 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: 42.826738+0000 mon.smithi058 (mon.0) 76 2021-11-20T05:29:43.976 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: : audit [DBG] from='mgr.14144 172.21.15.58:0/3887114449' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:29:43.976 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:29:43.976 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: -20T05:29:42.828412+0000 mon.smithi058 2021-11-20T05:29:43.977 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: (mon.0) 77 : audit [DBG] from='mgr.14144 172.21.15.58:0/3887114449' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:29:43.977 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: audit 2021-11-20T05:29:43.977 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: 2021-11-20T05:29:42.843774 2021-11-20T05:29:43.978 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 78 : audit 2021-11-20T05:29:43.978 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: [INF] from='mgr.14144 172.21.15.58:0/3887114449' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:29:43.978 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: audit 2021-11-20T 2021-11-20T05:29:43.978 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: 05:29:42.848010+0000 mon.smithi058 (mon. 2021-11-20T05:29:43.979 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:43 smithi058 conmon[33537]: 0) 79 : audit [INF] from='mgr.14144 172.21.15.58:0/3887114449' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:29:44.085 INFO:teuthology.orchestra.run.smithi058.stderr:Deploying grafana service with default placement... 2021-11-20T05:29:44.085 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2021-11-20T05:29:44.828 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.826+0000 7f3de8259700 1 Processor -- start 2021-11-20T05:29:44.829 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.826+0000 7f3de8259700 1 -- start start 2021-11-20T05:29:44.829 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.826+0000 7f3de8259700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3de00faae0 0x7f3de00faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:44.829 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.826+0000 7f3de8259700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de00fb480 con 0x7f3de00faae0 2021-11-20T05:29:44.829 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.827+0000 7f3de5ff5700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3de00faae0 0x7f3de00faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:44.830 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.827+0000 7f3de5ff5700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3de00faae0 0x7f3de00faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39730/0 (socket says 172.21.15.58:39730) 2021-11-20T05:29:44.830 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.827+0000 7f3de5ff5700 1 -- 172.21.15.58:0/4031277256 learned_addr learned my addr 172.21.15.58:0/4031277256 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:44.830 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.827+0000 7f3de5ff5700 1 -- 172.21.15.58:0/4031277256 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de00fb5c0 con 0x7f3de00faae0 2021-11-20T05:29:44.830 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.827+0000 7f3de5ff5700 1 --2- 172.21.15.58:0/4031277256 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3de00faae0 0x7f3de00faf00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f3dd000b5f0 tx=0x7f3dd0006ee0).ready entity=mon.0 client_cookie=9669f04db099c661 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:44.831 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.827+0000 7f3de4ff3700 1 -- 172.21.15.58:0/4031277256 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dd000d600 con 0x7f3de00faae0 2021-11-20T05:29:44.831 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.827+0000 7f3de4ff3700 1 -- 172.21.15.58:0/4031277256 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3dd000dc10 con 0x7f3de00faae0 2021-11-20T05:29:44.831 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.827+0000 7f3de4ff3700 1 -- 172.21.15.58:0/4031277256 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dd0011cd0 con 0x7f3de00faae0 2021-11-20T05:29:44.834 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.828+0000 7f3de8259700 1 -- 172.21.15.58:0/4031277256 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3de00faae0 msgr2=0x7f3de00faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:44.834 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.828+0000 7f3de8259700 1 --2- 172.21.15.58:0/4031277256 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3de00faae0 0x7f3de00faf00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f3dd000b5f0 tx=0x7f3dd0006ee0).stop 2021-11-20T05:29:44.835 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.828+0000 7f3de8259700 1 -- 172.21.15.58:0/4031277256 shutdown_connections 2021-11-20T05:29:44.835 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.828+0000 7f3de8259700 1 --2- 172.21.15.58:0/4031277256 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3de00faae0 0x7f3de00faf00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:44.835 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.828+0000 7f3de8259700 1 -- 172.21.15.58:0/4031277256 >> 172.21.15.58:0/4031277256 conn(0x7f3de00f6280 msgr2=0x7f3de00f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:44.835 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.829+0000 7f3de8259700 1 -- 172.21.15.58:0/4031277256 shutdown_connections 2021-11-20T05:29:44.836 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.829+0000 7f3de8259700 1 -- 172.21.15.58:0/4031277256 wait complete. 2021-11-20T05:29:44.836 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.829+0000 7f3de8259700 1 Processor -- start 2021-11-20T05:29:44.836 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.829+0000 7f3de8259700 1 -- start start 2021-11-20T05:29:44.836 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.829+0000 7f3de8259700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3de00faae0 0x7f3de010d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:44.837 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.829+0000 7f3de8259700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de006fc50 con 0x7f3de00faae0 2021-11-20T05:29:44.837 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.830+0000 7f3de5ff5700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3de00faae0 0x7f3de010d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:44.837 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.830+0000 7f3de5ff5700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3de00faae0 0x7f3de010d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39732/0 (socket says 172.21.15.58:39732) 2021-11-20T05:29:44.837 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.830+0000 7f3de5ff5700 1 -- 172.21.15.58:0/3314232168 learned_addr learned my addr 172.21.15.58:0/3314232168 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:44.837 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.830+0000 7f3de5ff5700 1 -- 172.21.15.58:0/3314232168 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd000b200 con 0x7f3de00faae0 2021-11-20T05:29:44.838 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.830+0000 7f3de5ff5700 1 --2- 172.21.15.58:0/3314232168 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3de00faae0 0x7f3de010d790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f3dd000dfe0 tx=0x7f3dd0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:44.838 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.830+0000 7f3dce7fc700 1 -- 172.21.15.58:0/3314232168 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dd0011df0 con 0x7f3de00faae0 2021-11-20T05:29:44.838 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.830+0000 7f3dce7fc700 1 -- 172.21.15.58:0/3314232168 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3dd001b8c0 con 0x7f3de00faae0 2021-11-20T05:29:44.839 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.830+0000 7f3dce7fc700 1 -- 172.21.15.58:0/3314232168 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dd001a840 con 0x7f3de00faae0 2021-11-20T05:29:44.839 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.830+0000 7f3de8259700 1 -- 172.21.15.58:0/3314232168 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de006fe50 con 0x7f3de00faae0 2021-11-20T05:29:44.839 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.830+0000 7f3de8259700 1 -- 172.21.15.58:0/3314232168 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de010de40 con 0x7f3de00faae0 2021-11-20T05:29:44.839 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.831+0000 7f3dce7fc700 1 -- 172.21.15.58:0/3314232168 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f3dd000f070 con 0x7f3de00faae0 2021-11-20T05:29:44.840 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.832+0000 7f3dce7fc700 1 --2- 172.21.15.58:0/3314232168 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f3dc802fcf0 0x7f3dc80321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:44.840 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.832+0000 7f3dce7fc700 1 -- 172.21.15.58:0/3314232168 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f3dd0040f70 con 0x7f3de00faae0 2021-11-20T05:29:44.840 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.832+0000 7f3de57f4700 1 --2- 172.21.15.58:0/3314232168 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f3dc802fcf0 0x7f3dc80321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:44.841 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.832+0000 7f3de8259700 1 -- 172.21.15.58:0/3314232168 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de0045ad0 con 0x7f3de00faae0 2021-11-20T05:29:44.841 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.832+0000 7f3de57f4700 1 --2- 172.21.15.58:0/3314232168 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f3dc802fcf0 0x7f3dc80321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f3ddc0096a0 tx=0x7f3ddc006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:44.841 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:44.836+0000 7f3dce7fc700 1 -- 172.21.15.58:0/3314232168 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3dd0016070 con 0x7f3de00faae0 2021-11-20T05:29:44.969 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:44 smithi058 conmon[33537]: cluster 2021-11-20T05:29:43.690916+0000 mon.smithi058 ( 2021-11-20T05:29:44.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:44 smithi058 conmon[33537]: mon.0) 80 : cluster [DBG] mgrmap e11: smithi058.zagdlk(active, since 1.05696s) 2021-11-20T05:29:44.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:44 smithi058 conmon[33537]: cephadm 2021-11-20T05:29:43.692734+0000 mgr.smithi058.zagdlk (mgr.14144) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2021-11-20T05:29:44.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:44 smithi058 conmon[33537]: audit 2021-11-20T05:29:43.695361+0000 mon.smithi058 (mon.0) 81 : audit [INF] from='mgr.14144 172.21.15.58:0/3887114449' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:45.162 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:45.162+0000 7f3dce7fc700 1 -- 172.21.15.58:0/3314232168 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f3dd001b410 con 0x7f3de00faae0 2021-11-20T05:29:45.190 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:45.189+0000 7f3de8259700 1 -- 172.21.15.58:0/3314232168 --> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f3de00fc9a0 con 0x7f3dc802fcf0 2021-11-20T05:29:45.195 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:45.194+0000 7f3dce7fc700 1 -- 172.21.15.58:0/3314232168 <== mgr.14144 v2:172.21.15.58:6800/2422532259 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f3de00fc9a0 con 0x7f3dc802fcf0 2021-11-20T05:29:45.195 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: Scheduled grafana update... 2021-11-20T05:29:45.199 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:45.196+0000 7f3de8259700 1 -- 172.21.15.58:0/3314232168 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f3dc802fcf0 msgr2=0x7f3dc80321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:45.199 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:45.196+0000 7f3de8259700 1 --2- 172.21.15.58:0/3314232168 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f3dc802fcf0 0x7f3dc80321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f3ddc0096a0 tx=0x7f3ddc006b40).stop 2021-11-20T05:29:45.200 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:45.196+0000 7f3de8259700 1 -- 172.21.15.58:0/3314232168 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3de00faae0 msgr2=0x7f3de010d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:45.200 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:45.196+0000 7f3de8259700 1 --2- 172.21.15.58:0/3314232168 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3de00faae0 0x7f3de010d790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f3dd000dfe0 tx=0x7f3dd0006ee0).stop 2021-11-20T05:29:45.200 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:45.196+0000 7f3de8259700 1 -- 172.21.15.58:0/3314232168 shutdown_connections 2021-11-20T05:29:45.200 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:45.196+0000 7f3de8259700 1 --2- 172.21.15.58:0/3314232168 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f3dc802fcf0 0x7f3dc80321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:45.200 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:45.196+0000 7f3de8259700 1 --2- 172.21.15.58:0/3314232168 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3de00faae0 0x7f3de010d790 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:45.201 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:45.196+0000 7f3de8259700 1 -- 172.21.15.58:0/3314232168 >> 172.21.15.58:0/3314232168 conn(0x7f3de00f6280 msgr2=0x7f3de00f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:45.201 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:45.196+0000 7f3de8259700 1 -- 172.21.15.58:0/3314232168 shutdown_connections 2021-11-20T05:29:45.201 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:45.196+0000 7f3de8259700 1 -- 172.21.15.58:0/3314232168 wait complete. 2021-11-20T05:29:45.575 INFO:teuthology.orchestra.run.smithi058.stderr:Deploying node-exporter service with default placement... 2021-11-20T05:29:45.575 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2021-11-20T05:29:45.881 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:45 smithi058 conmon[33537]: audit 2021-11-20T05:29:44.710679+0000 mon.smithi058 2021-11-20T05:29:45.882 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:45 smithi058 conmon[33537]: (mon.0) 82 : audit [INF] from='mgr.14144 172.21.15.58:0/3887114449' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:45.882 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:45 smithi058 conmon[33537]: cluster 2021-11-20T05:29:45.165044+0000 mon.smithi058 (mon.0) 83 : 2021-11-20T05:29:45.882 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:45 smithi058 conmon[33537]: cluster [DBG] mgrmap e12: smithi058.zagdlk(active, since 2s) 2021-11-20T05:29:45.882 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:45 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:29:45.883 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:45 smithi058 conmon[33537]: -20T05:29:45.191884+0000 mgr.smithi058.zagdlk (mgr.14144) 3 : audit [DBG] 2021-11-20T05:29:45.883 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:45 smithi058 conmon[33537]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:29:45.883 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:45 smithi058 conmon[33537]: cephadm 2021-11-20T05:29:45.883 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:45 smithi058 conmon[33537]: 2021-11-20T05:29:45 2021-11-20T05:29:45.883 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:45 smithi058 conmon[33537]: .192525+0000 mgr.smithi058.zagdlk (mgr.14144) 4 : 2021-11-20T05:29:45.884 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:45 smithi058 conmon[33537]: cephadm [INF] Saving service grafana spec with placement count:1 2021-11-20T05:29:45.884 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:45 smithi058 conmon[33537]: audit 2021-11- 2021-11-20T05:29:45.884 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:45 smithi058 conmon[33537]: 20T05:29:45.195687+0000 2021-11-20T05:29:45.884 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:45 smithi058 conmon[33537]: mon.smithi058 (mon.0) 84 : audit [INF] 2021-11-20T05:29:45.884 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:45 smithi058 conmon[33537]: from='mgr.14144 172.21.15.58:0/3887114449' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:47.355 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.351+0000 7f7981876700 1 Processor -- start 2021-11-20T05:29:47.356 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.352+0000 7f7981876700 1 -- start start 2021-11-20T05:29:47.356 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.352+0000 7f7981876700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f797c0fc0c0 0x7f797c0fc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:47.356 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.352+0000 7f7981876700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c0fc9d0 con 0x7f797c0fc0c0 2021-11-20T05:29:47.357 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.352+0000 7f797affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f797c0fc0c0 0x7f797c0fc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:47.357 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.352+0000 7f797affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f797c0fc0c0 0x7f797c0fc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39736/0 (socket says 172.21.15.58:39736) 2021-11-20T05:29:47.357 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.352+0000 7f797affd700 1 -- 172.21.15.58:0/3312101607 learned_addr learned my addr 172.21.15.58:0/3312101607 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:47.357 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.352+0000 7f797affd700 1 -- 172.21.15.58:0/3312101607 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797c0fcb10 con 0x7f797c0fc0c0 2021-11-20T05:29:47.358 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.352+0000 7f797affd700 1 --2- 172.21.15.58:0/3312101607 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f797c0fc0c0 0x7f797c0fc4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f797000b870 tx=0x7f7970006f70).ready entity=mon.0 client_cookie=eba94bb1c0af79ac server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:47.358 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.353+0000 7f7979ffb700 1 -- 172.21.15.58:0/3312101607 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f797000da20 con 0x7f797c0fc0c0 2021-11-20T05:29:47.358 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.353+0000 7f7979ffb700 1 -- 172.21.15.58:0/3312101607 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7970010470 con 0x7f797c0fc0c0 2021-11-20T05:29:47.358 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.353+0000 7f7979ffb700 1 -- 172.21.15.58:0/3312101607 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f797000f5b0 con 0x7f797c0fc0c0 2021-11-20T05:29:47.358 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.353+0000 7f7981876700 1 -- 172.21.15.58:0/3312101607 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f797c0fc0c0 msgr2=0x7f797c0fc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:47.359 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.353+0000 7f7981876700 1 --2- 172.21.15.58:0/3312101607 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f797c0fc0c0 0x7f797c0fc4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f797000b870 tx=0x7f7970006f70).stop 2021-11-20T05:29:47.359 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.353+0000 7f7981876700 1 -- 172.21.15.58:0/3312101607 shutdown_connections 2021-11-20T05:29:47.359 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.353+0000 7f7981876700 1 --2- 172.21.15.58:0/3312101607 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f797c0fc0c0 0x7f797c0fc4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:47.359 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.353+0000 7f7981876700 1 -- 172.21.15.58:0/3312101607 >> 172.21.15.58:0/3312101607 conn(0x7f797c0f5a80 msgr2=0x7f797c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:47.360 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.353+0000 7f7981876700 1 -- 172.21.15.58:0/3312101607 shutdown_connections 2021-11-20T05:29:47.360 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.354+0000 7f7981876700 1 -- 172.21.15.58:0/3312101607 wait complete. 2021-11-20T05:29:47.360 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.354+0000 7f7981876700 1 Processor -- start 2021-11-20T05:29:47.360 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.354+0000 7f7981876700 1 -- start start 2021-11-20T05:29:47.361 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.354+0000 7f7981876700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f797c0fc0c0 0x7f797c1087e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:47.361 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.354+0000 7f7981876700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c108cd0 con 0x7f797c0fc0c0 2021-11-20T05:29:47.361 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.355+0000 7f797affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f797c0fc0c0 0x7f797c1087e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:47.361 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.355+0000 7f797affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f797c0fc0c0 0x7f797c1087e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39738/0 (socket says 172.21.15.58:39738) 2021-11-20T05:29:47.362 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.355+0000 7f797affd700 1 -- 172.21.15.58:0/2837787563 learned_addr learned my addr 172.21.15.58:0/2837787563 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:47.362 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.355+0000 7f797affd700 1 -- 172.21.15.58:0/2837787563 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797000b480 con 0x7f797c0fc0c0 2021-11-20T05:29:47.362 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.355+0000 7f797affd700 1 --2- 172.21.15.58:0/2837787563 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f797c0fc0c0 0x7f797c1087e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f797000dad0 tx=0x7f7970006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:47.362 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.355+0000 7f79637fe700 1 -- 172.21.15.58:0/2837787563 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f797000f3d0 con 0x7f797c0fc0c0 2021-11-20T05:29:47.364 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.355+0000 7f79637fe700 1 -- 172.21.15.58:0/2837787563 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f797000fce0 con 0x7f797c0fc0c0 2021-11-20T05:29:47.364 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.355+0000 7f7981876700 1 -- 172.21.15.58:0/2837787563 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f797c108ed0 con 0x7f797c0fc0c0 2021-11-20T05:29:47.365 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.355+0000 7f79637fe700 1 -- 172.21.15.58:0/2837787563 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7970019e00 con 0x7f797c0fc0c0 2021-11-20T05:29:47.365 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.355+0000 7f7981876700 1 -- 172.21.15.58:0/2837787563 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f797c1092e0 con 0x7f797c0fc0c0 2021-11-20T05:29:47.365 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.356+0000 7f79637fe700 1 -- 172.21.15.58:0/2837787563 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f797002b060 con 0x7f797c0fc0c0 2021-11-20T05:29:47.365 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.356+0000 7f79637fe700 1 --2- 172.21.15.58:0/2837787563 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f796802feb0 0x7f7968032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:47.365 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.356+0000 7f79637fe700 1 -- 172.21.15.58:0/2837787563 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f7970044010 con 0x7f797c0fc0c0 2021-11-20T05:29:47.366 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.357+0000 7f797a7fc700 1 --2- 172.21.15.58:0/2837787563 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f796802feb0 0x7f7968032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:47.366 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.357+0000 7f7981876700 1 -- 172.21.15.58:0/2837787563 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f797c059100 con 0x7f797c0fc0c0 2021-11-20T05:29:47.366 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.357+0000 7f797a7fc700 1 --2- 172.21.15.58:0/2837787563 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f796802feb0 0x7f7968032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f796c0096a0 tx=0x7f796c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:47.366 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.361+0000 7f79637fe700 1 -- 172.21.15.58:0/2837787563 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7970014080 con 0x7f797c0fc0c0 2021-11-20T05:29:47.700 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.699+0000 7f7981876700 1 -- 172.21.15.58:0/2837787563 --> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f797c102670 con 0x7f796802feb0 2021-11-20T05:29:47.705 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.704+0000 7f79637fe700 1 -- 172.21.15.58:0/2837787563 <== mgr.14144 v2:172.21.15.58:6800/2422532259 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f797c102670 con 0x7f796802feb0 2021-11-20T05:29:47.706 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2021-11-20T05:29:47.707 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.706+0000 7f7981876700 1 -- 172.21.15.58:0/2837787563 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f796802feb0 msgr2=0x7f7968032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:47.708 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.706+0000 7f7981876700 1 --2- 172.21.15.58:0/2837787563 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f796802feb0 0x7f7968032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f796c0096a0 tx=0x7f796c006b40).stop 2021-11-20T05:29:47.708 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.706+0000 7f7981876700 1 -- 172.21.15.58:0/2837787563 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f797c0fc0c0 msgr2=0x7f797c1087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:47.708 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.706+0000 7f7981876700 1 --2- 172.21.15.58:0/2837787563 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f797c0fc0c0 0x7f797c1087e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f797000dad0 tx=0x7f7970006f70).stop 2021-11-20T05:29:47.708 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.706+0000 7f7981876700 1 -- 172.21.15.58:0/2837787563 shutdown_connections 2021-11-20T05:29:47.709 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.706+0000 7f7981876700 1 --2- 172.21.15.58:0/2837787563 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f796802feb0 0x7f7968032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:47.709 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.706+0000 7f7981876700 1 --2- 172.21.15.58:0/2837787563 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f797c0fc0c0 0x7f797c1087e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:47.709 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.706+0000 7f7981876700 1 -- 172.21.15.58:0/2837787563 >> 172.21.15.58:0/2837787563 conn(0x7f797c0f5a80 msgr2=0x7f797c0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:47.710 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.706+0000 7f7981876700 1 -- 172.21.15.58:0/2837787563 shutdown_connections 2021-11-20T05:29:47.710 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:47.706+0000 7f7981876700 1 -- 172.21.15.58:0/2837787563 wait complete. 2021-11-20T05:29:48.117 INFO:teuthology.orchestra.run.smithi058.stderr:Deploying alertmanager service with default placement... 2021-11-20T05:29:48.118 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2021-11-20T05:29:48.649 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.646+0000 7eff36bc7700 1 Processor -- start 2021-11-20T05:29:48.650 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.646+0000 7eff36bc7700 1 -- start start 2021-11-20T05:29:48.650 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.646+0000 7eff36bc7700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7eff300faad0 0x7eff300faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:48.650 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.646+0000 7eff36bc7700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff300fb3e0 con 0x7eff300faad0 2021-11-20T05:29:48.650 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.646+0000 7eff34963700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7eff300faad0 0x7eff300faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:48.651 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.646+0000 7eff34963700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7eff300faad0 0x7eff300faef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39742/0 (socket says 172.21.15.58:39742) 2021-11-20T05:29:48.651 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.646+0000 7eff34963700 1 -- 172.21.15.58:0/1589726923 learned_addr learned my addr 172.21.15.58:0/1589726923 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:48.651 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.646+0000 7eff34963700 1 -- 172.21.15.58:0/1589726923 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff300fb520 con 0x7eff300faad0 2021-11-20T05:29:48.651 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.647+0000 7eff34963700 1 --2- 172.21.15.58:0/1589726923 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7eff300faad0 0x7eff300faef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7eff2400b5f0 tx=0x7eff24006ee0).ready entity=mon.0 client_cookie=2e25431268188d7b server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:48.652 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.647+0000 7eff2f7fe700 1 -- 172.21.15.58:0/1589726923 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2400d600 con 0x7eff300faad0 2021-11-20T05:29:48.652 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.647+0000 7eff2f7fe700 1 -- 172.21.15.58:0/1589726923 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff2400dc10 con 0x7eff300faad0 2021-11-20T05:29:48.652 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.647+0000 7eff2f7fe700 1 -- 172.21.15.58:0/1589726923 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff24011cd0 con 0x7eff300faad0 2021-11-20T05:29:48.653 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.647+0000 7eff36bc7700 1 -- 172.21.15.58:0/1589726923 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7eff300faad0 msgr2=0x7eff300faef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:48.653 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.647+0000 7eff36bc7700 1 --2- 172.21.15.58:0/1589726923 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7eff300faad0 0x7eff300faef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7eff2400b5f0 tx=0x7eff24006ee0).stop 2021-11-20T05:29:48.653 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.647+0000 7eff36bc7700 1 -- 172.21.15.58:0/1589726923 shutdown_connections 2021-11-20T05:29:48.653 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.647+0000 7eff36bc7700 1 --2- 172.21.15.58:0/1589726923 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7eff300faad0 0x7eff300faef0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:48.654 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.647+0000 7eff36bc7700 1 -- 172.21.15.58:0/1589726923 >> 172.21.15.58:0/1589726923 conn(0x7eff300f6310 msgr2=0x7eff300f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:48.654 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.648+0000 7eff36bc7700 1 -- 172.21.15.58:0/1589726923 shutdown_connections 2021-11-20T05:29:48.654 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.648+0000 7eff36bc7700 1 -- 172.21.15.58:0/1589726923 wait complete. 2021-11-20T05:29:48.654 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.648+0000 7eff36bc7700 1 Processor -- start 2021-11-20T05:29:48.655 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.648+0000 7eff36bc7700 1 -- start start 2021-11-20T05:29:48.655 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.649+0000 7eff36bc7700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7eff300faad0 0x7eff30108780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:48.655 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.649+0000 7eff36bc7700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff30108c70 con 0x7eff300faad0 2021-11-20T05:29:48.655 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.649+0000 7eff34963700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7eff300faad0 0x7eff30108780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:48.655 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.649+0000 7eff34963700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7eff300faad0 0x7eff30108780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39744/0 (socket says 172.21.15.58:39744) 2021-11-20T05:29:48.656 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.649+0000 7eff34963700 1 -- 172.21.15.58:0/2886911961 learned_addr learned my addr 172.21.15.58:0/2886911961 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:48.657 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.649+0000 7eff34963700 1 -- 172.21.15.58:0/2886911961 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2400b200 con 0x7eff300faad0 2021-11-20T05:29:48.658 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.649+0000 7eff34963700 1 --2- 172.21.15.58:0/2886911961 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7eff300faad0 0x7eff30108780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7eff2400e040 tx=0x7eff24006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:48.658 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.649+0000 7eff2d7fa700 1 -- 172.21.15.58:0/2886911961 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2401b410 con 0x7eff300faad0 2021-11-20T05:29:48.658 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.649+0000 7eff2d7fa700 1 -- 172.21.15.58:0/2886911961 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff2401ba20 con 0x7eff300faad0 2021-11-20T05:29:48.658 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.649+0000 7eff36bc7700 1 -- 172.21.15.58:0/2886911961 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff30108e70 con 0x7eff300faad0 2021-11-20T05:29:48.659 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.649+0000 7eff2d7fa700 1 -- 172.21.15.58:0/2886911961 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2401a910 con 0x7eff300faad0 2021-11-20T05:29:48.659 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.649+0000 7eff36bc7700 1 -- 172.21.15.58:0/2886911961 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff30109280 con 0x7eff300faad0 2021-11-20T05:29:48.659 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.651+0000 7eff2d7fa700 1 -- 172.21.15.58:0/2886911961 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7eff2400f070 con 0x7eff300faad0 2021-11-20T05:29:48.660 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.651+0000 7eff36bc7700 1 -- 172.21.15.58:0/2886911961 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff30045ad0 con 0x7eff300faad0 2021-11-20T05:29:48.660 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.651+0000 7eff2d7fa700 1 --2- 172.21.15.58:0/2886911961 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7eff140320a0 0x7eff14034560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:48.660 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.651+0000 7eff2d7fa700 1 -- 172.21.15.58:0/2886911961 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7eff24041280 con 0x7eff300faad0 2021-11-20T05:29:48.660 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.651+0000 7eff2ffff700 1 --2- 172.21.15.58:0/2886911961 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7eff140320a0 0x7eff14034560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:48.661 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.652+0000 7eff2ffff700 1 --2- 172.21.15.58:0/2886911961 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7eff140320a0 0x7eff14034560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7eff1c0096a0 tx=0x7eff1c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:48.661 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.655+0000 7eff2d7fa700 1 -- 172.21.15.58:0/2886911961 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff24019cd0 con 0x7eff300faad0 2021-11-20T05:29:48.969 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:48 smithi058 conmon[33537]: audit 2021-11-20T05:29:47.702811+0000 mgr.smithi058.zagdlk (mgr.14144) 5 2021-11-20T05:29:48.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:48 smithi058 conmon[33537]: : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:29:48.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:48 smithi058 conmon[33537]: cephadm 2021-11-20T05:29:47.703498+0000 mgr.smithi058.zagdlk (mgr.14144) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2021-11-20T05:29:48.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:48 smithi058 conmon[33537]: audit 2021-11-20T05:29:47.705635+0000 mon.smithi058 (mon.0) 85 : audit [INF] from='mgr.14144 172.21.15.58:0/3887114449' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:48.996 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.995+0000 7eff36bc7700 1 -- 172.21.15.58:0/2886911961 --> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7eff30109bc0 con 0x7eff140320a0 2021-11-20T05:29:48.999 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:48.999+0000 7eff2d7fa700 1 -- 172.21.15.58:0/2886911961 <== mgr.14144 v2:172.21.15.58:6800/2422532259 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7eff30109bc0 con 0x7eff140320a0 2021-11-20T05:29:49.000 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2021-11-20T05:29:49.002 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.001+0000 7eff36bc7700 1 -- 172.21.15.58:0/2886911961 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7eff140320a0 msgr2=0x7eff14034560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:49.002 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.001+0000 7eff36bc7700 1 --2- 172.21.15.58:0/2886911961 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7eff140320a0 0x7eff14034560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7eff1c0096a0 tx=0x7eff1c006b40).stop 2021-11-20T05:29:49.003 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.001+0000 7eff36bc7700 1 -- 172.21.15.58:0/2886911961 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7eff300faad0 msgr2=0x7eff30108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:49.003 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.001+0000 7eff36bc7700 1 --2- 172.21.15.58:0/2886911961 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7eff300faad0 0x7eff30108780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7eff2400e040 tx=0x7eff24006ee0).stop 2021-11-20T05:29:49.003 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.001+0000 7eff36bc7700 1 -- 172.21.15.58:0/2886911961 shutdown_connections 2021-11-20T05:29:49.003 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.001+0000 7eff36bc7700 1 --2- 172.21.15.58:0/2886911961 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7eff140320a0 0x7eff14034560 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:49.004 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.001+0000 7eff36bc7700 1 --2- 172.21.15.58:0/2886911961 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7eff300faad0 0x7eff30108780 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:49.004 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.001+0000 7eff36bc7700 1 -- 172.21.15.58:0/2886911961 >> 172.21.15.58:0/2886911961 conn(0x7eff300f6310 msgr2=0x7eff300ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:49.004 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.001+0000 7eff36bc7700 1 -- 172.21.15.58:0/2886911961 shutdown_connections 2021-11-20T05:29:49.004 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.001+0000 7eff36bc7700 1 -- 172.21.15.58:0/2886911961 wait complete. 2021-11-20T05:29:49.355 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2021-11-20T05:29:49.828 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.825+0000 7f4be016a700 1 Processor -- start 2021-11-20T05:29:49.828 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.825+0000 7f4be016a700 1 -- start start 2021-11-20T05:29:49.828 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.825+0000 7f4be016a700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4bd80fd2c0 0x7f4bd80fd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:49.828 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.825+0000 7f4be016a700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd80fdbd0 con 0x7f4bd80fd2c0 2021-11-20T05:29:49.829 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.826+0000 7f4bddf06700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4bd80fd2c0 0x7f4bd80fd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:49.829 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.826+0000 7f4bddf06700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4bd80fd2c0 0x7f4bd80fd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39748/0 (socket says 172.21.15.58:39748) 2021-11-20T05:29:49.829 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.826+0000 7f4bddf06700 1 -- 172.21.15.58:0/1417258637 learned_addr learned my addr 172.21.15.58:0/1417258637 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:49.829 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.826+0000 7f4bddf06700 1 -- 172.21.15.58:0/1417258637 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd80fdd10 con 0x7f4bd80fd2c0 2021-11-20T05:29:49.830 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.826+0000 7f4bddf06700 1 --2- 172.21.15.58:0/1417258637 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4bd80fd2c0 0x7f4bd80fd6e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f4bd400b5f0 tx=0x7f4bd4006ee0).ready entity=mon.0 client_cookie=55b86529a312d905 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:49.830 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.826+0000 7f4bdcf04700 1 -- 172.21.15.58:0/1417258637 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bd400d600 con 0x7f4bd80fd2c0 2021-11-20T05:29:49.830 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.826+0000 7f4bdcf04700 1 -- 172.21.15.58:0/1417258637 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4bd400dc10 con 0x7f4bd80fd2c0 2021-11-20T05:29:49.830 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.826+0000 7f4bdcf04700 1 -- 172.21.15.58:0/1417258637 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bd4011cd0 con 0x7f4bd80fd2c0 2021-11-20T05:29:49.831 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.827+0000 7f4be016a700 1 -- 172.21.15.58:0/1417258637 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4bd80fd2c0 msgr2=0x7f4bd80fd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:49.831 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.827+0000 7f4be016a700 1 --2- 172.21.15.58:0/1417258637 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4bd80fd2c0 0x7f4bd80fd6e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f4bd400b5f0 tx=0x7f4bd4006ee0).stop 2021-11-20T05:29:49.834 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.827+0000 7f4be016a700 1 -- 172.21.15.58:0/1417258637 shutdown_connections 2021-11-20T05:29:49.834 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.827+0000 7f4be016a700 1 --2- 172.21.15.58:0/1417258637 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4bd80fd2c0 0x7f4bd80fd6e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:49.835 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.827+0000 7f4be016a700 1 -- 172.21.15.58:0/1417258637 >> 172.21.15.58:0/1417258637 conn(0x7f4bd80f5a80 msgr2=0x7f4bd80f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:49.835 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.827+0000 7f4be016a700 1 -- 172.21.15.58:0/1417258637 shutdown_connections 2021-11-20T05:29:49.835 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.827+0000 7f4be016a700 1 -- 172.21.15.58:0/1417258637 wait complete. 2021-11-20T05:29:49.835 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.828+0000 7f4be016a700 1 Processor -- start 2021-11-20T05:29:49.835 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.828+0000 7f4be016a700 1 -- start start 2021-11-20T05:29:49.836 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.828+0000 7f4be016a700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4bd80fd2c0 0x7f4bd80fcb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:49.836 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.828+0000 7f4be016a700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd80fb010 con 0x7f4bd80fd2c0 2021-11-20T05:29:49.836 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.828+0000 7f4bddf06700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4bd80fd2c0 0x7f4bd80fcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:49.836 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.828+0000 7f4bddf06700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4bd80fd2c0 0x7f4bd80fcb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39750/0 (socket says 172.21.15.58:39750) 2021-11-20T05:29:49.837 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.828+0000 7f4bddf06700 1 -- 172.21.15.58:0/2622527337 learned_addr learned my addr 172.21.15.58:0/2622527337 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:49.837 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.829+0000 7f4bddf06700 1 -- 172.21.15.58:0/2622527337 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd400b200 con 0x7f4bd80fd2c0 2021-11-20T05:29:49.837 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.829+0000 7f4bddf06700 1 --2- 172.21.15.58:0/2622527337 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4bd80fd2c0 0x7f4bd80fcb90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f4bd400dfb0 tx=0x7f4bd4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:49.837 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.829+0000 7f4bc67fc700 1 -- 172.21.15.58:0/2622527337 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bd4011df0 con 0x7f4bd80fd2c0 2021-11-20T05:29:49.837 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.829+0000 7f4bc67fc700 1 -- 172.21.15.58:0/2622527337 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4bd401b8c0 con 0x7f4bd80fd2c0 2021-11-20T05:29:49.838 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.829+0000 7f4be016a700 1 -- 172.21.15.58:0/2622527337 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd80fb1b0 con 0x7f4bd80fd2c0 2021-11-20T05:29:49.838 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.829+0000 7f4bc67fc700 1 -- 172.21.15.58:0/2622527337 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bd401a7c0 con 0x7f4bd80fd2c0 2021-11-20T05:29:49.838 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.829+0000 7f4be016a700 1 -- 172.21.15.58:0/2622527337 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd80fb5c0 con 0x7f4bd80fd2c0 2021-11-20T05:29:49.838 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.830+0000 7f4bc67fc700 1 -- 172.21.15.58:0/2622527337 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f4bd400f070 con 0x7f4bd80fd2c0 2021-11-20T05:29:49.839 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.830+0000 7f4be016a700 1 -- 172.21.15.58:0/2622527337 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd8045ad0 con 0x7f4bd80fd2c0 2021-11-20T05:29:49.839 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.830+0000 7f4bc67fc700 1 --2- 172.21.15.58:0/2622527337 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f4bc002fe60 0x7f4bc0032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:49.839 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.830+0000 7f4bc67fc700 1 -- 172.21.15.58:0/2622527337 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4bd4041090 con 0x7f4bd80fd2c0 2021-11-20T05:29:49.839 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.831+0000 7f4bdd705700 1 --2- 172.21.15.58:0/2622527337 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f4bc002fe60 0x7f4bc0032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:49.839 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.831+0000 7f4bdd705700 1 --2- 172.21.15.58:0/2622527337 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f4bc002fe60 0x7f4bc0032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f4bc80096a0 tx=0x7f4bc8006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:49.840 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:49.835+0000 7f4bc67fc700 1 -- 172.21.15.58:0/2622527337 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4bd4019d60 con 0x7f4bd80fd2c0 2021-11-20T05:29:50.122 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:50 smithi058 conmon[33537]: audit 2021-11-20T05:29:48.998014+0000 mgr.smithi058.zagdlk (mgr.14144) 7 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:29:50.122 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:50 smithi058 conmon[33537]: cephadm 2021-11-20T05:29:48.998560+0000 mgr.smithi058.zagdlk (mgr.14144) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2021-11-20T05:29:50.122 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:50 smithi058 conmon[33537]: audit 2021-11-20T05:29:49.000800+0000 mon.smithi058 (mon.0) 86 : audit [INF] from='mgr.14144 172.21.15.58:0/3887114449' entity='mgr.smithi058.zagdlk' 2021-11-20T05:29:50.123 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:50.121+0000 7f4be016a700 1 -- 172.21.15.58:0/2622527337 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f4bd80fa7a0 con 0x7f4bd80fd2c0 2021-11-20T05:29:50.124 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:50.124+0000 7f4bc67fc700 1 -- 172.21.15.58:0/2622527337 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f4bd4016070 con 0x7f4bd80fd2c0 2021-11-20T05:29:50.127 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:50.126+0000 7f4be016a700 1 -- 172.21.15.58:0/2622527337 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f4bc002fe60 msgr2=0x7f4bc0032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:50.127 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:50.126+0000 7f4be016a700 1 --2- 172.21.15.58:0/2622527337 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f4bc002fe60 0x7f4bc0032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f4bc80096a0 tx=0x7f4bc8006b40).stop 2021-11-20T05:29:50.127 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:50.126+0000 7f4be016a700 1 -- 172.21.15.58:0/2622527337 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4bd80fd2c0 msgr2=0x7f4bd80fcb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:50.128 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:50.126+0000 7f4be016a700 1 --2- 172.21.15.58:0/2622527337 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4bd80fd2c0 0x7f4bd80fcb90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f4bd400dfb0 tx=0x7f4bd4006ee0).stop 2021-11-20T05:29:50.129 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:50.126+0000 7f4be016a700 1 -- 172.21.15.58:0/2622527337 shutdown_connections 2021-11-20T05:29:50.129 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:50.126+0000 7f4be016a700 1 --2- 172.21.15.58:0/2622527337 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f4bc002fe60 0x7f4bc0032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:50.129 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:50.126+0000 7f4be016a700 1 --2- 172.21.15.58:0/2622527337 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4bd80fd2c0 0x7f4bd80fcb90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:50.129 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:50.126+0000 7f4be016a700 1 -- 172.21.15.58:0/2622527337 >> 172.21.15.58:0/2622527337 conn(0x7f4bd80f5a80 msgr2=0x7f4bd80ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:50.130 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:50.127+0000 7f4be016a700 1 -- 172.21.15.58:0/2622527337 shutdown_connections 2021-11-20T05:29:50.130 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:50.127+0000 7f4be016a700 1 -- 172.21.15.58:0/2622527337 wait complete. 2021-11-20T05:29:50.493 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2021-11-20T05:29:51.026 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.024+0000 7f956ba46700 1 Processor -- start 2021-11-20T05:29:51.026 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.024+0000 7f956ba46700 1 -- start start 2021-11-20T05:29:51.026 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.024+0000 7f956ba46700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f95640fd2a0 0x7f95640fd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:51.026 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.024+0000 7f956ba46700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95640fdbb0 con 0x7f95640fd2a0 2021-11-20T05:29:51.027 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.024+0000 7f95697e2700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f95640fd2a0 0x7f95640fd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:51.027 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.024+0000 7f95697e2700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f95640fd2a0 0x7f95640fd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39754/0 (socket says 172.21.15.58:39754) 2021-11-20T05:29:51.027 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.024+0000 7f95697e2700 1 -- 172.21.15.58:0/2877655196 learned_addr learned my addr 172.21.15.58:0/2877655196 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:51.027 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.025+0000 7f95697e2700 1 -- 172.21.15.58:0/2877655196 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95640fdcf0 con 0x7f95640fd2a0 2021-11-20T05:29:51.031 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.025+0000 7f95697e2700 1 --2- 172.21.15.58:0/2877655196 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f95640fd2a0 0x7f95640fd6c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f956000b5f0 tx=0x7f9560006ee0).ready entity=mon.0 client_cookie=d1b0ba4b8c6de3e7 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:51.031 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.025+0000 7f9553fff700 1 -- 172.21.15.58:0/2877655196 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f956000d600 con 0x7f95640fd2a0 2021-11-20T05:29:51.031 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.025+0000 7f9553fff700 1 -- 172.21.15.58:0/2877655196 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f956000dc10 con 0x7f95640fd2a0 2021-11-20T05:29:51.032 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.025+0000 7f9553fff700 1 -- 172.21.15.58:0/2877655196 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9560011cd0 con 0x7f95640fd2a0 2021-11-20T05:29:51.032 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.026+0000 7f956ba46700 1 -- 172.21.15.58:0/2877655196 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f95640fd2a0 msgr2=0x7f95640fd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:51.032 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.026+0000 7f956ba46700 1 --2- 172.21.15.58:0/2877655196 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f95640fd2a0 0x7f95640fd6c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f956000b5f0 tx=0x7f9560006ee0).stop 2021-11-20T05:29:51.032 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.026+0000 7f956ba46700 1 -- 172.21.15.58:0/2877655196 shutdown_connections 2021-11-20T05:29:51.033 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.026+0000 7f956ba46700 1 --2- 172.21.15.58:0/2877655196 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f95640fd2a0 0x7f95640fd6c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:51.033 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.026+0000 7f956ba46700 1 -- 172.21.15.58:0/2877655196 >> 172.21.15.58:0/2877655196 conn(0x7f95640f5a80 msgr2=0x7f95640f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:51.033 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.026+0000 7f956ba46700 1 -- 172.21.15.58:0/2877655196 shutdown_connections 2021-11-20T05:29:51.033 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.026+0000 7f956ba46700 1 -- 172.21.15.58:0/2877655196 wait complete. 2021-11-20T05:29:51.033 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.026+0000 7f956ba46700 1 Processor -- start 2021-11-20T05:29:51.034 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.027+0000 7f956ba46700 1 -- start start 2021-11-20T05:29:51.034 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.027+0000 7f956ba46700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f95640fd2a0 0x7f95640fcb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:51.034 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.027+0000 7f956ba46700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95640faff0 con 0x7f95640fd2a0 2021-11-20T05:29:51.034 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.027+0000 7f95697e2700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f95640fd2a0 0x7f95640fcb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:51.035 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.027+0000 7f95697e2700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f95640fd2a0 0x7f95640fcb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39756/0 (socket says 172.21.15.58:39756) 2021-11-20T05:29:51.035 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.027+0000 7f95697e2700 1 -- 172.21.15.58:0/3114079647 learned_addr learned my addr 172.21.15.58:0/3114079647 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:51.035 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.027+0000 7f95697e2700 1 -- 172.21.15.58:0/3114079647 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956000b200 con 0x7f95640fd2a0 2021-11-20T05:29:51.035 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.027+0000 7f95697e2700 1 --2- 172.21.15.58:0/3114079647 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f95640fd2a0 0x7f95640fcb70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f9560006cb0 tx=0x7f9560006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:51.035 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.027+0000 7f9551ffb700 1 -- 172.21.15.58:0/3114079647 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9560011df0 con 0x7f95640fd2a0 2021-11-20T05:29:51.036 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.027+0000 7f9551ffb700 1 -- 172.21.15.58:0/3114079647 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f956001b8c0 con 0x7f95640fd2a0 2021-11-20T05:29:51.036 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.028+0000 7f956ba46700 1 -- 172.21.15.58:0/3114079647 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95640fb190 con 0x7f95640fd2a0 2021-11-20T05:29:51.036 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.028+0000 7f9551ffb700 1 -- 172.21.15.58:0/3114079647 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f956001a930 con 0x7f95640fd2a0 2021-11-20T05:29:51.036 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.028+0000 7f956ba46700 1 -- 172.21.15.58:0/3114079647 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95640fb5a0 con 0x7f95640fd2a0 2021-11-20T05:29:51.037 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.029+0000 7f9551ffb700 1 -- 172.21.15.58:0/3114079647 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f956000f070 con 0x7f95640fd2a0 2021-11-20T05:29:51.037 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.029+0000 7f9551ffb700 1 --2- 172.21.15.58:0/3114079647 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f954c02fe60 0x7f954c032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:51.037 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.029+0000 7f9551ffb700 1 -- 172.21.15.58:0/3114079647 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f9560040fb0 con 0x7f95640fd2a0 2021-11-20T05:29:51.037 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.029+0000 7f956ba46700 1 -- 172.21.15.58:0/3114079647 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9564045ad0 con 0x7f95640fd2a0 2021-11-20T05:29:51.037 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.029+0000 7f9568fe1700 1 --2- 172.21.15.58:0/3114079647 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f954c02fe60 0x7f954c032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:51.038 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.029+0000 7f9568fe1700 1 --2- 172.21.15.58:0/3114079647 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f954c02fe60 0x7f954c032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f95540096a0 tx=0x7f9554006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:51.038 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.033+0000 7f9551ffb700 1 -- 172.21.15.58:0/3114079647 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9560016070 con 0x7f95640fd2a0 2021-11-20T05:29:51.320 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:51 smithi058 conmon[33537]: audit 2021-11-20T05:29:50.125840+0000 mon.smithi058 (mon.0) 87 : audit [INF] from='client.? 172.21.15.58:0/2622527337' entity='client.admin' 2021-11-20T05:29:51.321 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.320+0000 7f956ba46700 1 -- 172.21.15.58:0/3114079647 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f95640fa7c0 con 0x7f95640fd2a0 2021-11-20T05:29:51.323 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.323+0000 7f9551ffb700 1 -- 172.21.15.58:0/3114079647 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7f9560024410 con 0x7f95640fd2a0 2021-11-20T05:29:51.326 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.326+0000 7f956ba46700 1 -- 172.21.15.58:0/3114079647 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f954c02fe60 msgr2=0x7f954c032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:51.326 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.326+0000 7f956ba46700 1 --2- 172.21.15.58:0/3114079647 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f954c02fe60 0x7f954c032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f95540096a0 tx=0x7f9554006b40).stop 2021-11-20T05:29:51.327 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.326+0000 7f956ba46700 1 -- 172.21.15.58:0/3114079647 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f95640fd2a0 msgr2=0x7f95640fcb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:51.327 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.326+0000 7f956ba46700 1 --2- 172.21.15.58:0/3114079647 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f95640fd2a0 0x7f95640fcb70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f9560006cb0 tx=0x7f9560006ee0).stop 2021-11-20T05:29:51.327 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.326+0000 7f956ba46700 1 -- 172.21.15.58:0/3114079647 shutdown_connections 2021-11-20T05:29:51.327 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.326+0000 7f956ba46700 1 --2- 172.21.15.58:0/3114079647 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f954c02fe60 0x7f954c032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:51.328 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.326+0000 7f956ba46700 1 --2- 172.21.15.58:0/3114079647 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f95640fd2a0 0x7f95640fcb70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:51.328 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.326+0000 7f956ba46700 1 -- 172.21.15.58:0/3114079647 >> 172.21.15.58:0/3114079647 conn(0x7f95640f5a80 msgr2=0x7f95640ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:51.328 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.326+0000 7f956ba46700 1 -- 172.21.15.58:0/3114079647 shutdown_connections 2021-11-20T05:29:51.328 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:51.326+0000 7f956ba46700 1 -- 172.21.15.58:0/3114079647 wait complete. 2021-11-20T05:29:51.729 INFO:teuthology.orchestra.run.smithi058.stderr:Enabling the dashboard module... 2021-11-20T05:29:51.729 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2021-11-20T05:29:52.638 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:52 smithi058 conmon[33537]: audit 2021-11-20T05:29:51.324888+0000 mon.smithi058 (mon.0) 88 : audit [INF] from='client.? 172.21.15.58:0/3114079647' entity='client.admin' 2021-11-20T05:29:52.761 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.638+0000 7f1658331700 1 Processor -- start 2021-11-20T05:29:52.761 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.638+0000 7f1658331700 1 -- start start 2021-11-20T05:29:52.761 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.639+0000 7f1658331700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f16500fc210 0x7f16500fc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:52.762 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.639+0000 7f1658331700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16500fcb20 con 0x7f16500fc210 2021-11-20T05:29:52.762 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.639+0000 7f16560cd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f16500fc210 0x7f16500fc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:52.762 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.639+0000 7f16560cd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f16500fc210 0x7f16500fc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39760/0 (socket says 172.21.15.58:39760) 2021-11-20T05:29:52.762 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.639+0000 7f16560cd700 1 -- 172.21.15.58:0/3627083652 learned_addr learned my addr 172.21.15.58:0/3627083652 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:52.763 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.639+0000 7f16560cd700 1 -- 172.21.15.58:0/3627083652 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16500fcc60 con 0x7f16500fc210 2021-11-20T05:29:52.763 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.639+0000 7f16560cd700 1 --2- 172.21.15.58:0/3627083652 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f16500fc210 0x7f16500fc630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f164c00b5f0 tx=0x7f164c006ee0).ready entity=mon.0 client_cookie=b1adc1bbc65ff7ca server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:52.763 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.640+0000 7f16550cb700 1 -- 172.21.15.58:0/3627083652 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f164c00d600 con 0x7f16500fc210 2021-11-20T05:29:52.763 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.640+0000 7f16550cb700 1 -- 172.21.15.58:0/3627083652 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f164c00dc10 con 0x7f16500fc210 2021-11-20T05:29:52.763 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.640+0000 7f16550cb700 1 -- 172.21.15.58:0/3627083652 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f164c011cd0 con 0x7f16500fc210 2021-11-20T05:29:52.764 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.640+0000 7f1658331700 1 -- 172.21.15.58:0/3627083652 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f16500fc210 msgr2=0x7f16500fc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:52.764 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.640+0000 7f1658331700 1 --2- 172.21.15.58:0/3627083652 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f16500fc210 0x7f16500fc630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f164c00b5f0 tx=0x7f164c006ee0).stop 2021-11-20T05:29:52.764 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.640+0000 7f1658331700 1 -- 172.21.15.58:0/3627083652 shutdown_connections 2021-11-20T05:29:52.764 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.640+0000 7f1658331700 1 --2- 172.21.15.58:0/3627083652 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f16500fc210 0x7f16500fc630 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:52.765 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.640+0000 7f1658331700 1 -- 172.21.15.58:0/3627083652 >> 172.21.15.58:0/3627083652 conn(0x7f16500f5a80 msgr2=0x7f16500f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:52.765 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.640+0000 7f1658331700 1 -- 172.21.15.58:0/3627083652 shutdown_connections 2021-11-20T05:29:52.765 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.640+0000 7f1658331700 1 -- 172.21.15.58:0/3627083652 wait complete. 2021-11-20T05:29:52.765 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.641+0000 7f1658331700 1 Processor -- start 2021-11-20T05:29:52.765 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.641+0000 7f1658331700 1 -- start start 2021-11-20T05:29:52.766 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.641+0000 7f1658331700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f16500fc210 0x7f165010cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:52.766 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.641+0000 7f1658331700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165010d1d0 con 0x7f16500fc210 2021-11-20T05:29:52.766 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.641+0000 7f16560cd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f16500fc210 0x7f165010cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:52.766 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.641+0000 7f16560cd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f16500fc210 0x7f165010cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39762/0 (socket says 172.21.15.58:39762) 2021-11-20T05:29:52.767 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.641+0000 7f16560cd700 1 -- 172.21.15.58:0/3120197526 learned_addr learned my addr 172.21.15.58:0/3120197526 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:52.767 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.641+0000 7f16560cd700 1 -- 172.21.15.58:0/3120197526 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164c00b200 con 0x7f16500fc210 2021-11-20T05:29:52.767 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.642+0000 7f16560cd700 1 --2- 172.21.15.58:0/3120197526 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f16500fc210 0x7f165010cce0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f164c00dfb0 tx=0x7f164c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:52.767 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.642+0000 7f163effd700 1 -- 172.21.15.58:0/3120197526 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f164c011df0 con 0x7f16500fc210 2021-11-20T05:29:52.767 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.642+0000 7f163effd700 1 -- 172.21.15.58:0/3120197526 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f164c01b8c0 con 0x7f16500fc210 2021-11-20T05:29:52.768 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.642+0000 7f1658331700 1 -- 172.21.15.58:0/3120197526 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165010d3d0 con 0x7f16500fc210 2021-11-20T05:29:52.768 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.642+0000 7f163effd700 1 -- 172.21.15.58:0/3120197526 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f164c01a7c0 con 0x7f16500fc210 2021-11-20T05:29:52.768 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.642+0000 7f1658331700 1 -- 172.21.15.58:0/3120197526 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f165010d7e0 con 0x7f16500fc210 2021-11-20T05:29:52.768 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.643+0000 7f163effd700 1 -- 172.21.15.58:0/3120197526 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f164c00f070 con 0x7f16500fc210 2021-11-20T05:29:52.769 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.643+0000 7f163effd700 1 --2- 172.21.15.58:0/3120197526 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f163802feb0 0x7f1638032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:52.769 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.643+0000 7f163effd700 1 -- 172.21.15.58:0/3120197526 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f164c040f70 con 0x7f16500fc210 2021-11-20T05:29:52.769 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.643+0000 7f1658331700 1 -- 172.21.15.58:0/3120197526 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1650045ad0 con 0x7f16500fc210 2021-11-20T05:29:52.769 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.643+0000 7f16558cc700 1 --2- 172.21.15.58:0/3120197526 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f163802feb0 0x7f1638032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:52.769 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.643+0000 7f16558cc700 1 --2- 172.21.15.58:0/3120197526 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f163802feb0 0x7f1638032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f16400096a0 tx=0x7f1640006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:52.770 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:52.647+0000 7f163effd700 1 -- 172.21.15.58:0/3120197526 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f164c024410 con 0x7f16500fc210 2021-11-20T05:29:53.006 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:53.006+0000 7f1658331700 1 -- 172.21.15.58:0/3120197526 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f1650101de0 con 0x7f16500fc210 2021-11-20T05:29:53.326 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:53.326+0000 7f163effd700 1 -- 172.21.15.58:0/3120197526 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f164c010dd0 con 0x7f16500fc210 2021-11-20T05:29:53.327 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:53.327+0000 7f163effd700 1 -- 172.21.15.58:0/3120197526 <== mon.0 v2:172.21.15.58:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f164c017280 con 0x7f16500fc210 2021-11-20T05:29:53.332 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:53.329+0000 7f1658331700 1 -- 172.21.15.58:0/3120197526 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f163802feb0 msgr2=0x7f1638032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:53.332 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:53.329+0000 7f1658331700 1 --2- 172.21.15.58:0/3120197526 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f163802feb0 0x7f1638032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f16400096a0 tx=0x7f1640006b40).stop 2021-11-20T05:29:53.332 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:53.329+0000 7f1658331700 1 -- 172.21.15.58:0/3120197526 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f16500fc210 msgr2=0x7f165010cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:53.333 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:53.329+0000 7f1658331700 1 --2- 172.21.15.58:0/3120197526 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f16500fc210 0x7f165010cce0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f164c00dfb0 tx=0x7f164c006ee0).stop 2021-11-20T05:29:53.333 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:53.329+0000 7f1658331700 1 -- 172.21.15.58:0/3120197526 shutdown_connections 2021-11-20T05:29:53.333 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:53.329+0000 7f1658331700 1 --2- 172.21.15.58:0/3120197526 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f163802feb0 0x7f1638032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:53.333 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:53.330+0000 7f1658331700 1 --2- 172.21.15.58:0/3120197526 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f16500fc210 0x7f165010cce0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:53.334 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:53.330+0000 7f1658331700 1 -- 172.21.15.58:0/3120197526 >> 172.21.15.58:0/3120197526 conn(0x7f16500f5a80 msgr2=0x7f16500f7630 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:53.334 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:53.330+0000 7f1658331700 1 -- 172.21.15.58:0/3120197526 shutdown_connections 2021-11-20T05:29:53.334 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:53.330+0000 7f1658331700 1 -- 172.21.15.58:0/3120197526 wait complete. 2021-11-20T05:29:53.588 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:53 smithi058 conmon[33537]: audit 2021-11-20T05:29:53.007489+0000 mon.smithi058 (mon 2021-11-20T05:29:53.588 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:53 smithi058 conmon[33537]: .0) 89 : audit [INF] from='client.? 172.21.15.58:0/3120197526' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2021-11-20T05:29:53.767 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2021-11-20T05:29:54.306 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.303+0000 7fa6e6f1a700 1 Processor -- start 2021-11-20T05:29:54.306 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.303+0000 7fa6e6f1a700 1 -- start start 2021-11-20T05:29:54.307 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.303+0000 7fa6e6f1a700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa6e00faae0 0x7fa6e00faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:54.307 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.303+0000 7fa6e6f1a700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e00fb480 con 0x7fa6e00faae0 2021-11-20T05:29:54.307 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.303+0000 7fa6e4cb6700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa6e00faae0 0x7fa6e00faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:54.307 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.303+0000 7fa6e4cb6700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa6e00faae0 0x7fa6e00faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39776/0 (socket says 172.21.15.58:39776) 2021-11-20T05:29:54.308 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.303+0000 7fa6e4cb6700 1 -- 172.21.15.58:0/1533135940 learned_addr learned my addr 172.21.15.58:0/1533135940 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:54.308 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.304+0000 7fa6e4cb6700 1 -- 172.21.15.58:0/1533135940 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e00fb5c0 con 0x7fa6e00faae0 2021-11-20T05:29:54.308 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.304+0000 7fa6e4cb6700 1 --2- 172.21.15.58:0/1533135940 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa6e00faae0 0x7fa6e00faf00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fa6d400b5f0 tx=0x7fa6d4006ee0).ready entity=mon.0 client_cookie=5e3edcf9e9413589 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:54.309 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.304+0000 7fa6df7fe700 1 -- 172.21.15.58:0/1533135940 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6d400d600 con 0x7fa6e00faae0 2021-11-20T05:29:54.309 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.304+0000 7fa6df7fe700 1 -- 172.21.15.58:0/1533135940 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6d400dc10 con 0x7fa6e00faae0 2021-11-20T05:29:54.309 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.304+0000 7fa6df7fe700 1 -- 172.21.15.58:0/1533135940 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6d4011cd0 con 0x7fa6e00faae0 2021-11-20T05:29:54.309 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.305+0000 7fa6e6f1a700 1 -- 172.21.15.58:0/1533135940 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa6e00faae0 msgr2=0x7fa6e00faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:54.310 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.305+0000 7fa6e6f1a700 1 --2- 172.21.15.58:0/1533135940 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa6e00faae0 0x7fa6e00faf00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fa6d400b5f0 tx=0x7fa6d4006ee0).stop 2021-11-20T05:29:54.310 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.305+0000 7fa6e6f1a700 1 -- 172.21.15.58:0/1533135940 shutdown_connections 2021-11-20T05:29:54.310 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.305+0000 7fa6e6f1a700 1 --2- 172.21.15.58:0/1533135940 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa6e00faae0 0x7fa6e00faf00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:54.310 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.305+0000 7fa6e6f1a700 1 -- 172.21.15.58:0/1533135940 >> 172.21.15.58:0/1533135940 conn(0x7fa6e00f6280 msgr2=0x7fa6e00f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:54.311 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.305+0000 7fa6e6f1a700 1 -- 172.21.15.58:0/1533135940 shutdown_connections 2021-11-20T05:29:54.311 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.305+0000 7fa6e6f1a700 1 -- 172.21.15.58:0/1533135940 wait complete. 2021-11-20T05:29:54.311 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.305+0000 7fa6e6f1a700 1 Processor -- start 2021-11-20T05:29:54.311 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.305+0000 7fa6e6f1a700 1 -- start start 2021-11-20T05:29:54.311 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.306+0000 7fa6e6f1a700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa6e00faae0 0x7fa6e010d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:54.312 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.306+0000 7fa6e6f1a700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e006fcb0 con 0x7fa6e00faae0 2021-11-20T05:29:54.312 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.306+0000 7fa6e4cb6700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa6e00faae0 0x7fa6e010d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:54.312 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.306+0000 7fa6e4cb6700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa6e00faae0 0x7fa6e010d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39778/0 (socket says 172.21.15.58:39778) 2021-11-20T05:29:54.312 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.306+0000 7fa6e4cb6700 1 -- 172.21.15.58:0/3009214051 learned_addr learned my addr 172.21.15.58:0/3009214051 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:54.314 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.306+0000 7fa6e4cb6700 1 -- 172.21.15.58:0/3009214051 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d400b200 con 0x7fa6e00faae0 2021-11-20T05:29:54.314 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.306+0000 7fa6e4cb6700 1 --2- 172.21.15.58:0/3009214051 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa6e00faae0 0x7fa6e010d710 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fa6d4006800 tx=0x7fa6d4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:54.314 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.307+0000 7fa6dd7fa700 1 -- 172.21.15.58:0/3009214051 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6d4011df0 con 0x7fa6e00faae0 2021-11-20T05:29:54.315 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.307+0000 7fa6dd7fa700 1 -- 172.21.15.58:0/3009214051 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6d401b8c0 con 0x7fa6e00faae0 2021-11-20T05:29:54.315 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.307+0000 7fa6e6f1a700 1 -- 172.21.15.58:0/3009214051 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e006feb0 con 0x7fa6e00faae0 2021-11-20T05:29:54.315 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.307+0000 7fa6dd7fa700 1 -- 172.21.15.58:0/3009214051 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6d401a800 con 0x7fa6e00faae0 2021-11-20T05:29:54.315 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.307+0000 7fa6e6f1a700 1 -- 172.21.15.58:0/3009214051 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e010ddc0 con 0x7fa6e00faae0 2021-11-20T05:29:54.316 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.308+0000 7fa6dd7fa700 1 -- 172.21.15.58:0/3009214051 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fa6d400f070 con 0x7fa6e00faae0 2021-11-20T05:29:54.316 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.308+0000 7fa6e6f1a700 1 -- 172.21.15.58:0/3009214051 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e0045ad0 con 0x7fa6e00faae0 2021-11-20T05:29:54.316 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.308+0000 7fa6dd7fa700 1 --2- 172.21.15.58:0/3009214051 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7fa6c402ff00 0x7fa6c40323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:54.316 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.308+0000 7fa6dd7fa700 1 -- 172.21.15.58:0/3009214051 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fa6d4040f70 con 0x7fa6e00faae0 2021-11-20T05:29:54.316 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.308+0000 7fa6dffff700 1 -- 172.21.15.58:0/3009214051 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7fa6c402ff00 msgr2=0x7fa6c40323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.58:6800/2422532259 2021-11-20T05:29:54.317 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.308+0000 7fa6dffff700 1 --2- 172.21.15.58:0/3009214051 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7fa6c402ff00 0x7fa6c40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T05:29:54.317 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.312+0000 7fa6dd7fa700 1 -- 172.21.15.58:0/3009214051 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6d406d2b0 con 0x7fa6e00faae0 2021-11-20T05:29:54.508 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.509+0000 7fa6dffff700 1 -- 172.21.15.58:0/3009214051 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7fa6c402ff00 msgr2=0x7fa6c40323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.58:6800/2422532259 2021-11-20T05:29:54.509 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.509+0000 7fa6dffff700 1 --2- 172.21.15.58:0/3009214051 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7fa6c402ff00 0x7fa6c40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T05:29:54.673 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:54 smithi058 conmon[33537]: audit 2021-11-20T05:29:53.328411+0000 2021-11-20T05:29:54.674 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:54 smithi058 conmon[33537]: mon.smithi058 (mon.0) 90 : audit [INF] from='client.? 172.21.15.58:0/3120197526' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2021-11-20T05:29:54.674 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:54 smithi058 conmon[33537]: cluster 2021-11-20T05:29:53.328468+0000 2021-11-20T05:29:54.674 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:54 smithi058 conmon[33537]: mon.smithi058 (mon.0) 91 : cluster [DBG] mgrmap e13: smithi058.zagdlk(active, since 10s) 2021-11-20T05:29:54.676 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.673+0000 7fa6e6f1a700 1 -- 172.21.15.58:0/3009214051 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa6e00fc930 con 0x7fa6e00faae0 2021-11-20T05:29:54.677 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.674+0000 7fa6dd7fa700 1 -- 172.21.15.58:0/3009214051 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7fa6d4016070 con 0x7fa6e00faae0 2021-11-20T05:29:54.677 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: { 2021-11-20T05:29:54.677 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "epoch": 13, 2021-11-20T05:29:54.677 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "available": true, 2021-11-20T05:29:54.678 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "active_name": "smithi058.zagdlk", 2021-11-20T05:29:54.678 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "num_standby": 0 2021-11-20T05:29:54.678 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: } 2021-11-20T05:29:54.678 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.676+0000 7fa6e6f1a700 1 -- 172.21.15.58:0/3009214051 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7fa6c402ff00 msgr2=0x7fa6c40323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T05:29:54.679 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.676+0000 7fa6e6f1a700 1 --2- 172.21.15.58:0/3009214051 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7fa6c402ff00 0x7fa6c40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:29:54.679 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.676+0000 7fa6e6f1a700 1 -- 172.21.15.58:0/3009214051 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa6e00faae0 msgr2=0x7fa6e010d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:54.679 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.676+0000 7fa6e6f1a700 1 --2- 172.21.15.58:0/3009214051 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa6e00faae0 0x7fa6e010d710 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fa6d4006800 tx=0x7fa6d4006ee0).stop 2021-11-20T05:29:54.679 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.676+0000 7fa6e6f1a700 1 -- 172.21.15.58:0/3009214051 shutdown_connections 2021-11-20T05:29:54.679 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.676+0000 7fa6e6f1a700 1 --2- 172.21.15.58:0/3009214051 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7fa6c402ff00 0x7fa6c40323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:29:54.680 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.676+0000 7fa6e6f1a700 1 --2- 172.21.15.58:0/3009214051 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa6e00faae0 0x7fa6e010d710 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:54.680 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.676+0000 7fa6e6f1a700 1 -- 172.21.15.58:0/3009214051 >> 172.21.15.58:0/3009214051 conn(0x7fa6e00f6280 msgr2=0x7fa6e00f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:54.680 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.677+0000 7fa6e6f1a700 1 -- 172.21.15.58:0/3009214051 shutdown_connections 2021-11-20T05:29:54.680 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:54.677+0000 7fa6e6f1a700 1 -- 172.21.15.58:0/3009214051 wait complete. 2021-11-20T05:29:55.011 INFO:teuthology.orchestra.run.smithi058.stderr:Waiting for the mgr to restart... 2021-11-20T05:29:55.011 INFO:teuthology.orchestra.run.smithi058.stderr:Waiting for mgr epoch 13... 2021-11-20T05:29:55.011 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2021-11-20T05:29:55.548 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.546+0000 7f9b91261700 1 Processor -- start 2021-11-20T05:29:55.548 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.546+0000 7f9b91261700 1 -- start start 2021-11-20T05:29:55.550 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.547+0000 7f9b91261700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b8c06fcd0 0x7f9b8c0fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:55.550 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.547+0000 7f9b91261700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c0fed20 con 0x7f9b8c06fcd0 2021-11-20T05:29:55.550 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.547+0000 7f9b8affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b8c06fcd0 0x7f9b8c0fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:55.551 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.547+0000 7f9b8affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b8c06fcd0 0x7f9b8c0fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39786/0 (socket says 172.21.15.58:39786) 2021-11-20T05:29:55.551 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.547+0000 7f9b8affd700 1 -- 172.21.15.58:0/2010533786 learned_addr learned my addr 172.21.15.58:0/2010533786 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:55.551 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.547+0000 7f9b8affd700 1 -- 172.21.15.58:0/2010533786 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c0fee60 con 0x7f9b8c06fcd0 2021-11-20T05:29:55.551 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.547+0000 7f9b8affd700 1 --2- 172.21.15.58:0/2010533786 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b8c06fcd0 0x7f9b8c0fe7a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f9b7c00b5f0 tx=0x7f9b7c006ee0).ready entity=mon.0 client_cookie=36dc73cb2afb56e6 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:55.552 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.548+0000 7f9b89ffb700 1 -- 172.21.15.58:0/2010533786 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b7c00d600 con 0x7f9b8c06fcd0 2021-11-20T05:29:55.552 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.548+0000 7f9b89ffb700 1 -- 172.21.15.58:0/2010533786 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b7c00dc10 con 0x7f9b8c06fcd0 2021-11-20T05:29:55.552 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.548+0000 7f9b89ffb700 1 -- 172.21.15.58:0/2010533786 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b7c011cd0 con 0x7f9b8c06fcd0 2021-11-20T05:29:55.555 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.549+0000 7f9b91261700 1 -- 172.21.15.58:0/2010533786 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b8c06fcd0 msgr2=0x7f9b8c0fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:29:55.555 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.549+0000 7f9b91261700 1 --2- 172.21.15.58:0/2010533786 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b8c06fcd0 0x7f9b8c0fe7a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f9b7c00b5f0 tx=0x7f9b7c006ee0).stop 2021-11-20T05:29:55.555 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.549+0000 7f9b91261700 1 -- 172.21.15.58:0/2010533786 shutdown_connections 2021-11-20T05:29:55.556 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.549+0000 7f9b91261700 1 --2- 172.21.15.58:0/2010533786 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b8c06fcd0 0x7f9b8c0fe7a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:29:55.556 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.549+0000 7f9b91261700 1 -- 172.21.15.58:0/2010533786 >> 172.21.15.58:0/2010533786 conn(0x7f9b8c0f6280 msgr2=0x7f9b8c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:29:55.556 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.549+0000 7f9b91261700 1 -- 172.21.15.58:0/2010533786 shutdown_connections 2021-11-20T05:29:55.556 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.549+0000 7f9b91261700 1 -- 172.21.15.58:0/2010533786 wait complete. 2021-11-20T05:29:55.556 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.549+0000 7f9b91261700 1 Processor -- start 2021-11-20T05:29:55.557 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.549+0000 7f9b91261700 1 -- start start 2021-11-20T05:29:55.557 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.550+0000 7f9b91261700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b8c06fcd0 0x7f9b8c0fc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:55.557 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.550+0000 7f9b91261700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c0fa6a0 con 0x7f9b8c06fcd0 2021-11-20T05:29:55.557 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.550+0000 7f9b8affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b8c06fcd0 0x7f9b8c0fc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:29:55.558 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.550+0000 7f9b8affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b8c06fcd0 0x7f9b8c0fc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39788/0 (socket says 172.21.15.58:39788) 2021-11-20T05:29:55.558 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.550+0000 7f9b8affd700 1 -- 172.21.15.58:0/3075858232 learned_addr learned my addr 172.21.15.58:0/3075858232 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:29:55.558 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.550+0000 7f9b8affd700 1 -- 172.21.15.58:0/3075858232 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b7c00b200 con 0x7f9b8c06fcd0 2021-11-20T05:29:55.558 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.550+0000 7f9b8affd700 1 --2- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b8c06fcd0 0x7f9b8c0fc220 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f9b7c00b1d0 tx=0x7f9b7c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:29:55.558 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.550+0000 7f9b837fe700 1 -- 172.21.15.58:0/3075858232 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b7c011df0 con 0x7f9b8c06fcd0 2021-11-20T05:29:55.559 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.551+0000 7f9b837fe700 1 -- 172.21.15.58:0/3075858232 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b7c01b8c0 con 0x7f9b8c06fcd0 2021-11-20T05:29:55.559 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.551+0000 7f9b91261700 1 -- 172.21.15.58:0/3075858232 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b8c0fa840 con 0x7f9b8c06fcd0 2021-11-20T05:29:55.559 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.551+0000 7f9b837fe700 1 -- 172.21.15.58:0/3075858232 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b7c01a7b0 con 0x7f9b8c06fcd0 2021-11-20T05:29:55.559 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.551+0000 7f9b91261700 1 -- 172.21.15.58:0/3075858232 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b8c0fac50 con 0x7f9b8c06fcd0 2021-11-20T05:29:55.560 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.551+0000 7f9b837fe700 1 -- 172.21.15.58:0/3075858232 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f9b7c00f070 con 0x7f9b8c06fcd0 2021-11-20T05:29:55.560 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.552+0000 7f9b837fe700 1 --2- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f9b6c02feb0 0x7f9b6c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:29:55.560 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.552+0000 7f9b91261700 1 -- 172.21.15.58:0/3075858232 --> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b8c059100 con 0x7f9b6c02feb0 2021-11-20T05:29:55.560 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.552+0000 7f9b837fe700 1 -- 172.21.15.58:0/3075858232 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f9b7c0413b0 con 0x7f9b8c06fcd0 2021-11-20T05:29:55.561 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.552+0000 7f9b8a7fc700 1 -- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f9b6c02feb0 msgr2=0x7f9b6c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.58:6800/2422532259 2021-11-20T05:29:55.561 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.552+0000 7f9b8a7fc700 1 --2- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f9b6c02feb0 0x7f9b6c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T05:29:55.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:55 smithi058 conmon[33537]: audit 2021-11-20T05:29:54.675044+0000 2021-11-20T05:29:55.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:29:55 smithi058 conmon[33537]: mon.smithi058 (mon.0) 92 : audit [DBG] from='client.? 172.21.15.58:0/3009214051' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-11-20T05:29:55.752 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.753+0000 7f9b8a7fc700 1 -- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f9b6c02feb0 msgr2=0x7f9b6c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.58:6800/2422532259 2021-11-20T05:29:55.753 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:55.753+0000 7f9b8a7fc700 1 --2- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f9b6c02feb0 0x7f9b6c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T05:29:56.154 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:56.154+0000 7f9b8a7fc700 1 -- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f9b6c02feb0 msgr2=0x7f9b6c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.58:6800/2422532259 2021-11-20T05:29:56.154 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:56.154+0000 7f9b8a7fc700 1 --2- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f9b6c02feb0 0x7f9b6c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-20T05:29:56.954 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:56.955+0000 7f9b8a7fc700 1 -- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f9b6c02feb0 msgr2=0x7f9b6c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.58:6800/2422532259 2021-11-20T05:29:56.955 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:56.955+0000 7f9b8a7fc700 1 --2- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f9b6c02feb0 0x7f9b6c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-20T05:29:58.557 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:58.556+0000 7f9b8a7fc700 1 -- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f9b6c02feb0 msgr2=0x7f9b6c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.58:6800/2422532259 2021-11-20T05:29:58.558 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:29:58.556+0000 7f9b8a7fc700 1 --2- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f9b6c02feb0 0x7f9b6c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2021-11-20T05:30:00.166 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:00.165+0000 7f9b837fe700 1 -- 172.21.15.58:0/3075858232 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f9b7c0257e0 con 0x7f9b8c06fcd0 2021-11-20T05:30:00.166 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:00.165+0000 7f9b837fe700 1 -- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f9b6c02feb0 msgr2=0x7f9b6c032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T05:30:00.166 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:00.165+0000 7f9b837fe700 1 --2- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f9b6c02feb0 0x7f9b6c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:30:00.434 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:00 smithi058 conmon[33537]: cluster 2021-11-20T05:29:59.311688+0000 mon.smithi058 (mon.0) 93 : cluster [INF] Active manager daemon smithi058.zagdlk restarted 2021-11-20T05:30:00.434 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:00 smithi058 conmon[33537]: cluster 2021-11-20T05:29:59.312097+0000 mon.smithi058 (mon.0) 94 : cluster [INF] Activating manager daemon smithi058.zagdlk 2021-11-20T05:30:00.435 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:00 smithi058 conmon[33537]: cluster 2021-11-20T05:29:59.313787+0000 mon.smithi058 (mon.0) 95 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-11-20T05:30:00.435 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:00 smithi058 conmon[33537]: cluster 2021-11-20T05:30:00.167570+0000 mon.smithi058 (mon.0) 96 : cluster [DBG] mgrmap e14: smithi058.zagdlk(active, starting, since 0.85555s) 2021-11-20T05:30:00.435 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:00 smithi058 conmon[33537]: audit 2021-11-20T05:30:00.168699+0000 mon.smithi058 (mon.0) 97 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi058"}]: dispatch 2021-11-20T05:30:00.435 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:00 smithi058 conmon[33537]: audit 2021-11-20T05:30:00.169684+0000 mon.smithi058 (mon.0) 98 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mgr metadata", "who": "smithi058.zagdlk", "id": "smithi058.zagdlk"}]: dispatch 2021-11-20T05:30:00.436 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:00 smithi058 conmon[33537]: audit 2021-11-20T05:30:00.170560+0000 mon.smithi058 (mon.0) 99 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-20T05:30:00.436 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:00 smithi058 conmon[33537]: audit 2021-11-20T05:30:00.170842+0000 mon.smithi058 (mon.0) 100 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-20T05:30:00.436 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:00 smithi058 conmon[33537]: audit 2021-11-20T05:30:00.171126+0000 mon.smithi058 (mon.0) 101 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-20T05:30:00.437 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:00 smithi058 conmon[33537]: cluster 2021-11-20T05:30:00.186938+0000 mon.smithi058 (mon.0) 102 : 2021-11-20T05:30:00.437 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:00 smithi058 conmon[33537]: cluster [INF] Manager daemon smithi058.zagdlk is now available 2021-11-20T05:30:01.170 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.166+0000 7f9b837fe700 1 -- 172.21.15.58:0/3075858232 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f9b7c01ba30 con 0x7f9b8c06fcd0 2021-11-20T05:30:01.170 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.166+0000 7f9b837fe700 1 --2- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9b6c033990 0x7f9b6c035d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:01.171 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.166+0000 7f9b837fe700 1 -- 172.21.15.58:0/3075858232 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b8c059100 con 0x7f9b6c033990 2021-11-20T05:30:01.171 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.167+0000 7f9b8a7fc700 1 --2- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9b6c033990 0x7f9b6c035d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:01.171 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.167+0000 7f9b8a7fc700 1 --2- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9b6c033990 0x7f9b6c035d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f9b74006fd0 tx=0x7f9b74006cf0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:01.171 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.168+0000 7f9b837fe700 1 -- 172.21.15.58:0/3075858232 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f9b8c059100 con 0x7f9b6c033990 2021-11-20T05:30:01.178 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.178+0000 7f9b91261700 1 -- 172.21.15.58:0/3075858232 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f9b8c045ad0 con 0x7f9b6c033990 2021-11-20T05:30:01.179 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.178+0000 7f9b837fe700 1 -- 172.21.15.58:0/3075858232 <== mgr.14164 v2:172.21.15.58:6800/2097301011 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f9b8c045ad0 con 0x7f9b6c033990 2021-11-20T05:30:01.182 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: { 2021-11-20T05:30:01.183 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2021-11-20T05:30:01.183 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: "initialized": true 2021-11-20T05:30:01.184 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: } 2021-11-20T05:30:01.184 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.179+0000 7f9b91261700 1 -- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9b6c033990 msgr2=0x7f9b6c035d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:01.185 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.179+0000 7f9b91261700 1 --2- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9b6c033990 0x7f9b6c035d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f9b74006fd0 tx=0x7f9b74006cf0).stop 2021-11-20T05:30:01.185 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.179+0000 7f9b91261700 1 -- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b8c06fcd0 msgr2=0x7f9b8c0fc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:01.186 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.179+0000 7f9b91261700 1 --2- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b8c06fcd0 0x7f9b8c0fc220 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f9b7c00b1d0 tx=0x7f9b7c006ee0).stop 2021-11-20T05:30:01.186 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.179+0000 7f9b91261700 1 -- 172.21.15.58:0/3075858232 shutdown_connections 2021-11-20T05:30:01.187 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.179+0000 7f9b91261700 1 --2- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9b6c033990 0x7f9b6c035d80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:01.188 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.179+0000 7f9b91261700 1 --2- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9b8c06fcd0 0x7f9b8c0fc220 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:01.188 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.179+0000 7f9b91261700 1 --2- 172.21.15.58:0/3075858232 >> [v2:172.21.15.58:6800/2422532259,v1:172.21.15.58:6801/2422532259] conn(0x7f9b6c02feb0 0x7f9b6c032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:30:01.188 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.179+0000 7f9b91261700 1 -- 172.21.15.58:0/3075858232 >> 172.21.15.58:0/3075858232 conn(0x7f9b8c0f6280 msgr2=0x7f9b8c0f7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:01.189 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.179+0000 7f9b91261700 1 -- 172.21.15.58:0/3075858232 shutdown_connections 2021-11-20T05:30:01.189 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:01.179+0000 7f9b91261700 1 -- 172.21.15.58:0/3075858232 wait complete. 2021-11-20T05:30:01.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:01 smithi058 conmon[33537]: audit 2021-11-20T05:30:00.330802+0000 2021-11-20T05:30:01.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:01 smithi058 conmon[33537]: mon.smithi058 (mon.0) 103 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:30:01.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:01 smithi058 conmon[33537]: audit 2021-11-20T05:30:00.333266+0000 mon.smithi058 (mon.0) 104 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:30:01.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:01 smithi058 conmon[33537]: audit 2021-11-20T05:30:00.352260+0000 mon.smithi058 (mon.0) 105 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:30:01.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:01 smithi058 conmon[33537]: audit 2021-11-20T05 2021-11-20T05:30:01.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:01 smithi058 conmon[33537]: :30:00.355602+0000 mon.smithi058 ( 2021-11-20T05:30:01.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:01 smithi058 conmon[33537]: mon.0) 106 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:30:01.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:01 smithi058 conmon[33537]: cluster 2021-11-20T05:30:01.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:01 smithi058 conmon[33537]: 2021-11-20T05:30:01.169536+0000 mon.smithi058 (mon.0) 107 : cluster 2021-11-20T05:30:01.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:01 smithi058 conmon[33537]: [DBG] mgrmap e15: smithi058.zagdlk(active, since 1.85752s) 2021-11-20T05:30:01.571 INFO:teuthology.orchestra.run.smithi058.stderr:mgr epoch 13 is available 2021-11-20T05:30:01.571 INFO:teuthology.orchestra.run.smithi058.stderr:Generating a dashboard self-signed certificate... 2021-11-20T05:30:01.572 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2021-11-20T05:30:02.230 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.220+0000 7fd9d9d8f700 1 Processor -- start 2021-11-20T05:30:02.230 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.220+0000 7fd9d9d8f700 1 -- start start 2021-11-20T05:30:02.231 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.220+0000 7fd9d9d8f700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd9d406fc70 0x7fd9d40fe800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:02.231 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.220+0000 7fd9d9d8f700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d40fed80 con 0x7fd9d406fc70 2021-11-20T05:30:02.232 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.220+0000 7fd9d37fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd9d406fc70 0x7fd9d40fe800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:02.232 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.221+0000 7fd9d37fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd9d406fc70 0x7fd9d40fe800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39836/0 (socket says 172.21.15.58:39836) 2021-11-20T05:30:02.232 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.221+0000 7fd9d37fe700 1 -- 172.21.15.58:0/669140129 learned_addr learned my addr 172.21.15.58:0/669140129 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:30:02.233 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.221+0000 7fd9d37fe700 1 -- 172.21.15.58:0/669140129 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d40feec0 con 0x7fd9d406fc70 2021-11-20T05:30:02.233 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.221+0000 7fd9d37fe700 1 --2- 172.21.15.58:0/669140129 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd9d406fc70 0x7fd9d40fe800 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fd9c400b5f0 tx=0x7fd9c4006ee0).ready entity=mon.0 client_cookie=1acbe28b13dcfa51 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:02.233 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.221+0000 7fd9d2ffd700 1 -- 172.21.15.58:0/669140129 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9c400d600 con 0x7fd9d406fc70 2021-11-20T05:30:02.234 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.221+0000 7fd9d2ffd700 1 -- 172.21.15.58:0/669140129 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9c400dc10 con 0x7fd9d406fc70 2021-11-20T05:30:02.234 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.221+0000 7fd9d2ffd700 1 -- 172.21.15.58:0/669140129 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9c4011cd0 con 0x7fd9d406fc70 2021-11-20T05:30:02.235 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.222+0000 7fd9d9d8f700 1 -- 172.21.15.58:0/669140129 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd9d406fc70 msgr2=0x7fd9d40fe800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:02.235 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.222+0000 7fd9d9d8f700 1 --2- 172.21.15.58:0/669140129 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd9d406fc70 0x7fd9d40fe800 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fd9c400b5f0 tx=0x7fd9c4006ee0).stop 2021-11-20T05:30:02.235 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.223+0000 7fd9d9d8f700 1 -- 172.21.15.58:0/669140129 shutdown_connections 2021-11-20T05:30:02.236 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.223+0000 7fd9d9d8f700 1 --2- 172.21.15.58:0/669140129 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd9d406fc70 0x7fd9d40fe800 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:02.236 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.223+0000 7fd9d9d8f700 1 -- 172.21.15.58:0/669140129 >> 172.21.15.58:0/669140129 conn(0x7fd9d40f6280 msgr2=0x7fd9d40f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:02.236 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.223+0000 7fd9d9d8f700 1 -- 172.21.15.58:0/669140129 shutdown_connections 2021-11-20T05:30:02.236 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.223+0000 7fd9d9d8f700 1 -- 172.21.15.58:0/669140129 wait complete. 2021-11-20T05:30:02.237 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.223+0000 7fd9d9d8f700 1 Processor -- start 2021-11-20T05:30:02.237 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.223+0000 7fd9d9d8f700 1 -- start start 2021-11-20T05:30:02.237 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.224+0000 7fd9d9d8f700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd9d406fc70 0x7fd9d40fc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:02.238 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.224+0000 7fd9d9d8f700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d40fa700 con 0x7fd9d406fc70 2021-11-20T05:30:02.238 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.224+0000 7fd9d37fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd9d406fc70 0x7fd9d40fc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:02.238 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.224+0000 7fd9d37fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd9d406fc70 0x7fd9d40fc280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39838/0 (socket says 172.21.15.58:39838) 2021-11-20T05:30:02.239 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.224+0000 7fd9d37fe700 1 -- 172.21.15.58:0/2026799175 learned_addr learned my addr 172.21.15.58:0/2026799175 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:30:02.239 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.224+0000 7fd9d37fe700 1 -- 172.21.15.58:0/2026799175 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c400b200 con 0x7fd9d406fc70 2021-11-20T05:30:02.240 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.224+0000 7fd9d37fe700 1 --2- 172.21.15.58:0/2026799175 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd9d406fc70 0x7fd9d40fc280 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fd9c4006cb0 tx=0x7fd9c4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:02.240 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.224+0000 7fd9d0ff9700 1 -- 172.21.15.58:0/2026799175 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9c4011df0 con 0x7fd9d406fc70 2021-11-20T05:30:02.240 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.224+0000 7fd9d0ff9700 1 -- 172.21.15.58:0/2026799175 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9c401b8c0 con 0x7fd9d406fc70 2021-11-20T05:30:02.241 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.224+0000 7fd9d0ff9700 1 -- 172.21.15.58:0/2026799175 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9c401a930 con 0x7fd9d406fc70 2021-11-20T05:30:02.241 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.225+0000 7fd9d9d8f700 1 -- 172.21.15.58:0/2026799175 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d40fa8a0 con 0x7fd9d406fc70 2021-11-20T05:30:02.241 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.225+0000 7fd9d9d8f700 1 -- 172.21.15.58:0/2026799175 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d40facb0 con 0x7fd9d406fc70 2021-11-20T05:30:02.241 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.226+0000 7fd9d0ff9700 1 -- 172.21.15.58:0/2026799175 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fd9c400f070 con 0x7fd9d406fc70 2021-11-20T05:30:02.241 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.226+0000 7fd9d9d8f700 1 -- 172.21.15.58:0/2026799175 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d4045ad0 con 0x7fd9d406fc70 2021-11-20T05:30:02.242 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.226+0000 7fd9d0ff9700 1 --2- 172.21.15.58:0/2026799175 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd9b402fd70 0x7fd9b4032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:02.242 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.226+0000 7fd9d0ff9700 1 -- 172.21.15.58:0/2026799175 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9c4040f30 con 0x7fd9d406fc70 2021-11-20T05:30:02.242 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.226+0000 7fd9cbfff700 1 --2- 172.21.15.58:0/2026799175 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd9b402fd70 0x7fd9b4032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:02.242 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.226+0000 7fd9cbfff700 1 --2- 172.21.15.58:0/2026799175 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd9b402fd70 0x7fd9b4032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd9bc0096a0 tx=0x7fd9bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:02.243 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.231+0000 7fd9d0ff9700 1 -- 172.21.15.58:0/2026799175 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9c406d2b0 con 0x7fd9d406fc70 2021-11-20T05:30:02.316 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.316+0000 7fd9d0ff9700 1 -- 172.21.15.58:0/2026799175 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fd9c401aaf0 con 0x7fd9d406fc70 2021-11-20T05:30:02.540 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:02 smithi058 conmon[33537]: audit 2021-11-20T05:30:01.168652+0000 mgr.smithi058.zagdlk (mgr.14164) 1 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-11-20T05:30:02.540 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:02 smithi058 conmon[33537]: audit 2021-11-20T05:30:01.179562+0000 mgr.smithi058.zagdlk (mgr.14164) 2 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-11-20T05:30:02.540 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:02 smithi058 conmon[33537]: audit 2021-11-20T05:30:02.105031+0000 mon.smithi058 (mon.0) 108 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:02.541 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.540+0000 7fd9d9d8f700 1 -- 172.21.15.58:0/2026799175 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9d41049a0 con 0x7fd9b402fd70 2021-11-20T05:30:02.720 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.720+0000 7fd9d0ff9700 1 -- 172.21.15.58:0/2026799175 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fd9d41049a0 con 0x7fd9b402fd70 2021-11-20T05:30:02.720 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: Self-signed certificate created 2021-11-20T05:30:02.722 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.721+0000 7fd9d9d8f700 1 -- 172.21.15.58:0/2026799175 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd9b402fd70 msgr2=0x7fd9b4032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:02.722 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.721+0000 7fd9d9d8f700 1 --2- 172.21.15.58:0/2026799175 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd9b402fd70 0x7fd9b4032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd9bc0096a0 tx=0x7fd9bc006b40).stop 2021-11-20T05:30:02.723 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.721+0000 7fd9d9d8f700 1 -- 172.21.15.58:0/2026799175 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd9d406fc70 msgr2=0x7fd9d40fc280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:02.723 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.721+0000 7fd9d9d8f700 1 --2- 172.21.15.58:0/2026799175 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd9d406fc70 0x7fd9d40fc280 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fd9c4006cb0 tx=0x7fd9c4006ee0).stop 2021-11-20T05:30:02.723 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.722+0000 7fd9d9d8f700 1 -- 172.21.15.58:0/2026799175 shutdown_connections 2021-11-20T05:30:02.723 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.722+0000 7fd9d9d8f700 1 --2- 172.21.15.58:0/2026799175 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd9b402fd70 0x7fd9b4032230 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:02.724 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.722+0000 7fd9d9d8f700 1 --2- 172.21.15.58:0/2026799175 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd9d406fc70 0x7fd9d40fc280 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:02.724 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.722+0000 7fd9d9d8f700 1 -- 172.21.15.58:0/2026799175 >> 172.21.15.58:0/2026799175 conn(0x7fd9d40f6280 msgr2=0x7fd9d40f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:02.725 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.722+0000 7fd9d9d8f700 1 -- 172.21.15.58:0/2026799175 shutdown_connections 2021-11-20T05:30:02.725 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:02.722+0000 7fd9d9d8f700 1 -- 172.21.15.58:0/2026799175 wait complete. 2021-11-20T05:30:03.262 INFO:teuthology.orchestra.run.smithi058.stderr:Creating initial admin user... 2021-11-20T05:30:03.263 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8tpzosoi:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2021-11-20T05:30:03.533 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:03 smithi058 conmon[33537]: cluster 2021-11-20T05:30:02.318481+0000 2021-11-20T05:30:03.534 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:03 smithi058 conmon[33537]: mon.smithi058 (mon.0) 109 : cluster [DBG] mgrmap e16: smithi058.zagdlk(active, since 3s) 2021-11-20T05:30:03.535 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:03 smithi058 conmon[33537]: audit 2021-11-20T05:30:02.718729+0000 mon.smithi058 (mon.0) 110 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:03.535 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:03 smithi058 conmon[33537]: audit 2021-11-20T05:30:02.720588+0000 mon.smithi058 (mon.0) 111 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:03.986 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.985+0000 7f9e38248700 1 Processor -- start 2021-11-20T05:30:03.986 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.985+0000 7f9e38248700 1 -- start start 2021-11-20T05:30:03.987 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.986+0000 7f9e38248700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e3006fc30 0x7f9e30100c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:03.987 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.986+0000 7f9e38248700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e300fc6c0 con 0x7f9e3006fc30 2021-11-20T05:30:03.987 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.986+0000 7f9e35fe4700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e3006fc30 0x7f9e30100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:03.987 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.986+0000 7f9e35fe4700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e3006fc30 0x7f9e30100c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39842/0 (socket says 172.21.15.58:39842) 2021-11-20T05:30:03.987 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.986+0000 7f9e35fe4700 1 -- 172.21.15.58:0/1723990612 learned_addr learned my addr 172.21.15.58:0/1723990612 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:30:03.988 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.986+0000 7f9e35fe4700 1 -- 172.21.15.58:0/1723990612 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e30101130 con 0x7f9e3006fc30 2021-11-20T05:30:03.988 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.986+0000 7f9e35fe4700 1 --2- 172.21.15.58:0/1723990612 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e3006fc30 0x7f9e30100c40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f9e2c00b5f0 tx=0x7f9e2c006ee0).ready entity=mon.0 client_cookie=479671ceadba9fa server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:03.991 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.987+0000 7f9e34fe2700 1 -- 172.21.15.58:0/1723990612 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e2c00d600 con 0x7f9e3006fc30 2021-11-20T05:30:03.992 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.987+0000 7f9e34fe2700 1 -- 172.21.15.58:0/1723990612 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e2c00dc10 con 0x7f9e3006fc30 2021-11-20T05:30:03.992 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.987+0000 7f9e34fe2700 1 -- 172.21.15.58:0/1723990612 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e2c011cd0 con 0x7f9e3006fc30 2021-11-20T05:30:03.992 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.987+0000 7f9e38248700 1 -- 172.21.15.58:0/1723990612 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e3006fc30 msgr2=0x7f9e30100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:03.992 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.987+0000 7f9e38248700 1 --2- 172.21.15.58:0/1723990612 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e3006fc30 0x7f9e30100c40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f9e2c00b5f0 tx=0x7f9e2c006ee0).stop 2021-11-20T05:30:03.993 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.987+0000 7f9e38248700 1 -- 172.21.15.58:0/1723990612 shutdown_connections 2021-11-20T05:30:03.993 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.987+0000 7f9e38248700 1 --2- 172.21.15.58:0/1723990612 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e3006fc30 0x7f9e30100c40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:03.993 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.987+0000 7f9e38248700 1 -- 172.21.15.58:0/1723990612 >> 172.21.15.58:0/1723990612 conn(0x7f9e300f6280 msgr2=0x7f9e300f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:03.993 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.988+0000 7f9e38248700 1 -- 172.21.15.58:0/1723990612 shutdown_connections 2021-11-20T05:30:03.993 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.988+0000 7f9e38248700 1 -- 172.21.15.58:0/1723990612 wait complete. 2021-11-20T05:30:03.994 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.989+0000 7f9e38248700 1 Processor -- start 2021-11-20T05:30:03.994 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.989+0000 7f9e38248700 1 -- start start 2021-11-20T05:30:03.994 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.989+0000 7f9e38248700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e3006fc30 0x7f9e3010cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:03.994 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.989+0000 7f9e38248700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3010d1d0 con 0x7f9e3006fc30 2021-11-20T05:30:03.995 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.989+0000 7f9e35fe4700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e3006fc30 0x7f9e3010cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:03.995 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.989+0000 7f9e35fe4700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e3006fc30 0x7f9e3010cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39844/0 (socket says 172.21.15.58:39844) 2021-11-20T05:30:03.995 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.989+0000 7f9e35fe4700 1 -- 172.21.15.58:0/4188418287 learned_addr learned my addr 172.21.15.58:0/4188418287 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:30:03.995 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.989+0000 7f9e35fe4700 1 -- 172.21.15.58:0/4188418287 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2c00b200 con 0x7f9e3006fc30 2021-11-20T05:30:03.996 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.989+0000 7f9e35fe4700 1 --2- 172.21.15.58:0/4188418287 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e3006fc30 0x7f9e3010cce0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f9e2c00b1d0 tx=0x7f9e2c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:03.996 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.990+0000 7f9e1e7fc700 1 -- 172.21.15.58:0/4188418287 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e2c011df0 con 0x7f9e3006fc30 2021-11-20T05:30:03.996 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.990+0000 7f9e1e7fc700 1 -- 172.21.15.58:0/4188418287 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e2c01b8c0 con 0x7f9e3006fc30 2021-11-20T05:30:03.996 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.990+0000 7f9e38248700 1 -- 172.21.15.58:0/4188418287 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e3010d3d0 con 0x7f9e3006fc30 2021-11-20T05:30:03.996 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.990+0000 7f9e1e7fc700 1 -- 172.21.15.58:0/4188418287 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e2c01a7b0 con 0x7f9e3006fc30 2021-11-20T05:30:03.997 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.990+0000 7f9e38248700 1 -- 172.21.15.58:0/4188418287 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e3010d7e0 con 0x7f9e3006fc30 2021-11-20T05:30:03.997 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.991+0000 7f9e1e7fc700 1 -- 172.21.15.58:0/4188418287 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f9e2c00f070 con 0x7f9e3006fc30 2021-11-20T05:30:03.997 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.991+0000 7f9e1e7fc700 1 --2- 172.21.15.58:0/4188418287 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9e1802ff00 0x7f9e180323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:03.997 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.991+0000 7f9e1e7fc700 1 -- 172.21.15.58:0/4188418287 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e2c041480 con 0x7f9e3006fc30 2021-11-20T05:30:03.998 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.991+0000 7f9e357e3700 1 --2- 172.21.15.58:0/4188418287 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9e1802ff00 0x7f9e180323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:03.998 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.992+0000 7f9e38248700 1 -- 172.21.15.58:0/4188418287 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e30045ad0 con 0x7f9e3006fc30 2021-11-20T05:30:03.998 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.992+0000 7f9e357e3700 1 --2- 172.21.15.58:0/4188418287 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9e1802ff00 0x7f9e180323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f9e200096a0 tx=0x7f9e20006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:03.999 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:03.996+0000 7f9e1e7fc700 1 -- 172.21.15.58:0/4188418287 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e2c016030 con 0x7f9e3006fc30 2021-11-20T05:30:04.314 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:04.314+0000 7f9e38248700 1 -- 172.21.15.58:0/4188418287 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9e30058f20 con 0x7f9e1802ff00 2021-11-20T05:30:04.560 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:04.560+0000 7f9e1e7fc700 1 -- 172.21.15.58:0/4188418287 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f9e30058f20 con 0x7f9e1802ff00 2021-11-20T05:30:04.560 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$Pzm1sj2PdikLO928VO180uqSePo0eebfgl7h7tUcSKamOeVD.kdYy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1637386204, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2021-11-20T05:30:04.563 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:04.562+0000 7f9e38248700 1 -- 172.21.15.58:0/4188418287 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9e1802ff00 msgr2=0x7f9e180323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:04.563 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:04.562+0000 7f9e38248700 1 --2- 172.21.15.58:0/4188418287 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9e1802ff00 0x7f9e180323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f9e200096a0 tx=0x7f9e20006b40).stop 2021-11-20T05:30:04.564 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:04.562+0000 7f9e38248700 1 -- 172.21.15.58:0/4188418287 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e3006fc30 msgr2=0x7f9e3010cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:04.564 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:04.562+0000 7f9e38248700 1 --2- 172.21.15.58:0/4188418287 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e3006fc30 0x7f9e3010cce0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f9e2c00b1d0 tx=0x7f9e2c006ee0).stop 2021-11-20T05:30:04.564 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:04.562+0000 7f9e38248700 1 -- 172.21.15.58:0/4188418287 shutdown_connections 2021-11-20T05:30:04.564 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:04.562+0000 7f9e38248700 1 --2- 172.21.15.58:0/4188418287 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9e1802ff00 0x7f9e180323c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:04.564 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:04.562+0000 7f9e38248700 1 --2- 172.21.15.58:0/4188418287 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e3006fc30 0x7f9e3010cce0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:04.565 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:04.562+0000 7f9e38248700 1 -- 172.21.15.58:0/4188418287 >> 172.21.15.58:0/4188418287 conn(0x7f9e300f6280 msgr2=0x7f9e300f7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:04.565 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:04.562+0000 7f9e38248700 1 -- 172.21.15.58:0/4188418287 shutdown_connections 2021-11-20T05:30:04.565 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:04.562+0000 7f9e38248700 1 -- 172.21.15.58:0/4188418287 wait complete. 2021-11-20T05:30:04.585 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:04 smithi058 conmon[33537]: audit 2021-11-20T05:30:02.541611+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:30:04.586 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:04 smithi058 conmon[33537]: mgr.14164) 3 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:30:04.919 INFO:teuthology.orchestra.run.smithi058.stderr:Fetching dashboard port number... 2021-11-20T05:30:04.919 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi058 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpa4tw45_o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfivz_wyw:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2021-11-20T05:30:05.452 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.448+0000 7fab314f5700 1 Processor -- start 2021-11-20T05:30:05.452 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.448+0000 7fab314f5700 1 -- start start 2021-11-20T05:30:05.453 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.448+0000 7fab314f5700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab2c06fc10 0x7fab2c0fe860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:05.453 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.448+0000 7fab314f5700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c0fed50 con 0x7fab2c06fc10 2021-11-20T05:30:05.453 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.448+0000 7fab2affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab2c06fc10 0x7fab2c0fe860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:05.454 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.449+0000 7fab2affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab2c06fc10 0x7fab2c0fe860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39848/0 (socket says 172.21.15.58:39848) 2021-11-20T05:30:05.454 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.449+0000 7fab2affd700 1 -- 172.21.15.58:0/3258009211 learned_addr learned my addr 172.21.15.58:0/3258009211 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:30:05.454 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.449+0000 7fab2affd700 1 -- 172.21.15.58:0/3258009211 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c0fee90 con 0x7fab2c06fc10 2021-11-20T05:30:05.454 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.449+0000 7fab2affd700 1 --2- 172.21.15.58:0/3258009211 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab2c06fc10 0x7fab2c0fe860 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fab2000b5f0 tx=0x7fab20006ee0).ready entity=mon.0 client_cookie=ebd4079a2b31c48d server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:05.454 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.449+0000 7fab29ffb700 1 -- 172.21.15.58:0/3258009211 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab2000d600 con 0x7fab2c06fc10 2021-11-20T05:30:05.455 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.449+0000 7fab29ffb700 1 -- 172.21.15.58:0/3258009211 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab2000dc10 con 0x7fab2c06fc10 2021-11-20T05:30:05.455 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.449+0000 7fab29ffb700 1 -- 172.21.15.58:0/3258009211 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab20011d00 con 0x7fab2c06fc10 2021-11-20T05:30:05.456 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.450+0000 7fab314f5700 1 -- 172.21.15.58:0/3258009211 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab2c06fc10 msgr2=0x7fab2c0fe860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:05.456 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.450+0000 7fab314f5700 1 --2- 172.21.15.58:0/3258009211 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab2c06fc10 0x7fab2c0fe860 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fab2000b5f0 tx=0x7fab20006ee0).stop 2021-11-20T05:30:05.456 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.450+0000 7fab314f5700 1 -- 172.21.15.58:0/3258009211 shutdown_connections 2021-11-20T05:30:05.456 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.450+0000 7fab314f5700 1 --2- 172.21.15.58:0/3258009211 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab2c06fc10 0x7fab2c0fe860 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:05.457 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.450+0000 7fab314f5700 1 -- 172.21.15.58:0/3258009211 >> 172.21.15.58:0/3258009211 conn(0x7fab2c0f6280 msgr2=0x7fab2c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:05.457 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.450+0000 7fab314f5700 1 -- 172.21.15.58:0/3258009211 shutdown_connections 2021-11-20T05:30:05.457 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.450+0000 7fab314f5700 1 -- 172.21.15.58:0/3258009211 wait complete. 2021-11-20T05:30:05.457 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.451+0000 7fab314f5700 1 Processor -- start 2021-11-20T05:30:05.457 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.451+0000 7fab314f5700 1 -- start start 2021-11-20T05:30:05.458 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.451+0000 7fab314f5700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab2c06fc10 0x7fab2c0fc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:05.458 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.451+0000 7fab314f5700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c0fa760 con 0x7fab2c06fc10 2021-11-20T05:30:05.458 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.451+0000 7fab2affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab2c06fc10 0x7fab2c0fc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:05.458 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.451+0000 7fab2affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab2c06fc10 0x7fab2c0fc2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39850/0 (socket says 172.21.15.58:39850) 2021-11-20T05:30:05.459 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.451+0000 7fab2affd700 1 -- 172.21.15.58:0/2845899079 learned_addr learned my addr 172.21.15.58:0/2845899079 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:30:05.460 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.452+0000 7fab2affd700 1 -- 172.21.15.58:0/2845899079 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2000b200 con 0x7fab2c06fc10 2021-11-20T05:30:05.460 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.452+0000 7fab2affd700 1 --2- 172.21.15.58:0/2845899079 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab2c06fc10 0x7fab2c0fc2e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fab2000e040 tx=0x7fab20006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:05.460 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.452+0000 7fab137fe700 1 -- 172.21.15.58:0/2845899079 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab20011e20 con 0x7fab2c06fc10 2021-11-20T05:30:05.461 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.452+0000 7fab314f5700 1 -- 172.21.15.58:0/2845899079 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab2c0fa900 con 0x7fab2c06fc10 2021-11-20T05:30:05.461 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.452+0000 7fab137fe700 1 -- 172.21.15.58:0/2845899079 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab2001b8c0 con 0x7fab2c06fc10 2021-11-20T05:30:05.461 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.452+0000 7fab137fe700 1 -- 172.21.15.58:0/2845899079 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab2001a790 con 0x7fab2c06fc10 2021-11-20T05:30:05.461 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.452+0000 7fab314f5700 1 -- 172.21.15.58:0/2845899079 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2c0fad10 con 0x7fab2c06fc10 2021-11-20T05:30:05.462 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.453+0000 7fab137fe700 1 -- 172.21.15.58:0/2845899079 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fab2000f070 con 0x7fab2c06fc10 2021-11-20T05:30:05.462 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.453+0000 7fab314f5700 1 -- 172.21.15.58:0/2845899079 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab0c004fa0 con 0x7fab2c06fc10 2021-11-20T05:30:05.462 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.453+0000 7fab137fe700 1 --2- 172.21.15.58:0/2845899079 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fab1802ff00 0x7fab180323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:05.462 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.454+0000 7fab137fe700 1 -- 172.21.15.58:0/2845899079 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fab200420e0 con 0x7fab2c06fc10 2021-11-20T05:30:05.462 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.454+0000 7fab2a7fc700 1 --2- 172.21.15.58:0/2845899079 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fab1802ff00 0x7fab180323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:05.463 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.454+0000 7fab2a7fc700 1 --2- 172.21.15.58:0/2845899079 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fab1802ff00 0x7fab180323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fab1c0096a0 tx=0x7fab1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:05.463 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.458+0000 7fab137fe700 1 -- 172.21.15.58:0/2845899079 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab200107d0 con 0x7fab2c06fc10 2021-11-20T05:30:05.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:05 smithi058 conmon[33537]: audit 2021-11-20T05:30:04.316326+0000 2021-11-20T05:30:05.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:05 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 4 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:30:05.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:05 smithi058 conmon[33537]: audit 2021-11-20T05:30 2021-11-20T05:30:05.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:05 smithi058 conmon[33537]: :04.561082+0000 mon.smithi058 (mon.0) 112 : audit 2021-11-20T05:30:05.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:05 smithi058 conmon[33537]: [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:05.745 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.745+0000 7fab314f5700 1 -- 172.21.15.58:0/2845899079 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fab0c005bf0 con 0x7fab2c06fc10 2021-11-20T05:30:05.745 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.745+0000 7fab137fe700 1 -- 172.21.15.58:0/2845899079 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7fab20016030 con 0x7fab2c06fc10 2021-11-20T05:30:05.745 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 8443 2021-11-20T05:30:05.747 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.747+0000 7fab314f5700 1 -- 172.21.15.58:0/2845899079 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fab1802ff00 msgr2=0x7fab180323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:05.748 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.747+0000 7fab314f5700 1 --2- 172.21.15.58:0/2845899079 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fab1802ff00 0x7fab180323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fab1c0096a0 tx=0x7fab1c006b40).stop 2021-11-20T05:30:05.748 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.747+0000 7fab314f5700 1 -- 172.21.15.58:0/2845899079 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab2c06fc10 msgr2=0x7fab2c0fc2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:05.748 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.747+0000 7fab314f5700 1 --2- 172.21.15.58:0/2845899079 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab2c06fc10 0x7fab2c0fc2e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fab2000e040 tx=0x7fab20006ee0).stop 2021-11-20T05:30:05.748 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.747+0000 7fab314f5700 1 -- 172.21.15.58:0/2845899079 shutdown_connections 2021-11-20T05:30:05.749 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.747+0000 7fab314f5700 1 --2- 172.21.15.58:0/2845899079 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fab1802ff00 0x7fab180323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:05.749 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.747+0000 7fab314f5700 1 --2- 172.21.15.58:0/2845899079 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab2c06fc10 0x7fab2c0fc2e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:05.749 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.747+0000 7fab314f5700 1 -- 172.21.15.58:0/2845899079 >> 172.21.15.58:0/2845899079 conn(0x7fab2c0f6280 msgr2=0x7fab2c0f7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:05.749 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.747+0000 7fab314f5700 1 -- 172.21.15.58:0/2845899079 shutdown_connections 2021-11-20T05:30:05.750 INFO:teuthology.orchestra.run.smithi058.stderr:/usr/bin/ceph: 2021-11-20T05:30:05.747+0000 7fab314f5700 1 -- 172.21.15.58:0/2845899079 wait complete. 2021-11-20T05:30:06.155 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-20T05:30:06.163 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: disabled 2021-11-20T05:30:06.164 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: systemctl is-active firewalld.service 2021-11-20T05:30:06.170 INFO:teuthology.orchestra.run.smithi058.stderr:systemctl: inactive 2021-11-20T05:30:06.170 INFO:teuthology.orchestra.run.smithi058.stderr:firewalld.service is not enabled 2021-11-20T05:30:06.171 INFO:teuthology.orchestra.run.smithi058.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2021-11-20T05:30:06.174 INFO:teuthology.orchestra.run.smithi058.stderr:Ceph Dashboard is now available at: 2021-11-20T05:30:06.174 INFO:teuthology.orchestra.run.smithi058.stderr: 2021-11-20T05:30:06.174 INFO:teuthology.orchestra.run.smithi058.stderr: URL: https://smithi058.front.sepia.ceph.com:8443/ 2021-11-20T05:30:06.174 INFO:teuthology.orchestra.run.smithi058.stderr: User: admin 2021-11-20T05:30:06.175 INFO:teuthology.orchestra.run.smithi058.stderr: Password: da8wnfcuma 2021-11-20T05:30:06.175 INFO:teuthology.orchestra.run.smithi058.stderr: 2021-11-20T05:30:06.175 INFO:teuthology.orchestra.run.smithi058.stderr:You can access the Ceph CLI with: 2021-11-20T05:30:06.175 INFO:teuthology.orchestra.run.smithi058.stderr: 2021-11-20T05:30:06.176 INFO:teuthology.orchestra.run.smithi058.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2021-11-20T05:30:06.176 INFO:teuthology.orchestra.run.smithi058.stderr: 2021-11-20T05:30:06.176 INFO:teuthology.orchestra.run.smithi058.stderr:Please consider enabling telemetry to help improve Ceph: 2021-11-20T05:30:06.176 INFO:teuthology.orchestra.run.smithi058.stderr: 2021-11-20T05:30:06.176 INFO:teuthology.orchestra.run.smithi058.stderr: ceph telemetry on 2021-11-20T05:30:06.177 INFO:teuthology.orchestra.run.smithi058.stderr: 2021-11-20T05:30:06.177 INFO:teuthology.orchestra.run.smithi058.stderr:For more information see: 2021-11-20T05:30:06.177 INFO:teuthology.orchestra.run.smithi058.stderr: 2021-11-20T05:30:06.177 INFO:teuthology.orchestra.run.smithi058.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2021-11-20T05:30:06.178 INFO:teuthology.orchestra.run.smithi058.stderr: 2021-11-20T05:30:06.178 INFO:teuthology.orchestra.run.smithi058.stderr:Bootstrap complete. 2021-11-20T05:30:06.178 INFO:teuthology.orchestra.run.smithi058.stderr:Releasing lock 140284633807560 on /run/cephadm/b33f8c4a-49c2-11ec-8c2c-001a4aab830c.lock 2021-11-20T05:30:06.178 INFO:teuthology.orchestra.run.smithi058.stderr:Lock 140284633807560 released on /run/cephadm/b33f8c4a-49c2-11ec-8c2c-001a4aab830c.lock 2021-11-20T05:30:06.208 INFO:tasks.cephadm:Fetching config... 2021-11-20T05:30:06.208 DEBUG:teuthology.orchestra.run.smithi058:> set -ex 2021-11-20T05:30:06.209 DEBUG:teuthology.orchestra.run.smithi058:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2021-11-20T05:30:06.273 INFO:tasks.cephadm:Fetching client.admin keyring... 2021-11-20T05:30:06.273 DEBUG:teuthology.orchestra.run.smithi058:> set -ex 2021-11-20T05:30:06.274 DEBUG:teuthology.orchestra.run.smithi058:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2021-11-20T05:30:06.329 INFO:tasks.cephadm:Fetching mon keyring... 2021-11-20T05:30:06.330 DEBUG:teuthology.orchestra.run.smithi058:> set -ex 2021-11-20T05:30:06.331 DEBUG:teuthology.orchestra.run.smithi058:> sudo dd if=/var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/keyring of=/dev/stdout 2021-11-20T05:30:06.375 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:06 smithi058 conmon[33537]: audit 2021-11-20T05:30:05.746368 2021-11-20T05:30:06.376 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:06 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 113 : audit [DBG] from='client.? 172.21.15.58:0/2845899079' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2021-11-20T05:30:06.397 INFO:tasks.cephadm:Fetching pub ssh key... 2021-11-20T05:30:06.397 DEBUG:teuthology.orchestra.run.smithi058:> set -ex 2021-11-20T05:30:06.398 DEBUG:teuthology.orchestra.run.smithi058:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2021-11-20T05:30:06.455 INFO:tasks.cephadm:Installing pub ssh key for root users... 2021-11-20T05:30:06.456 DEBUG:teuthology.orchestra.run.smithi058:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC86XmLpBDHobQdn3NGWif/qFSOW3QeM6e79vZ/ZCb2YHk00xy4YVhc+XXrE3Spq4TALoFcEge2K0yPCQVITrzG8kYfkUQiEI2J8iTh7PGOVonOWSNbPCldnjXyJJK0NYloqunrKaBlbCyjsJMQpGV4GOx6zvzWtxnPN9bvmEMDqSSF0HdcF+oMJf31bcjFJRqcYy1tUId1uf4RdfN1nCEcyrcHipSgT8ki+GoShA8faayD/v3PhmverG2YSNyxvWfQ5bfK9vPncOPs0wukrY39EKdy7I7CmpMCWGigy9+0zYBXAeW/4lpiHsy6FKp/WLuRkidsZm4+ohTIJ8jba9vJD1Ko6x67pV86qKmtnt889QtakunwswYAkmb5Z9oE7mG5AuSiDqRxR93uyf1fU0EiHdJwx9r/8Lnh3s+N2UvevjNatLT1yVO7BvuDbf+MvIhqnj6CH1Qrj+4kWEJCzskb/m9d0bxSR2ZL1PHbVTXGz5VgWcVMGqsIdq4yhVypl2k= ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-11-20T05:30:06.532 INFO:teuthology.orchestra.run.smithi058.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC86XmLpBDHobQdn3NGWif/qFSOW3QeM6e79vZ/ZCb2YHk00xy4YVhc+XXrE3Spq4TALoFcEge2K0yPCQVITrzG8kYfkUQiEI2J8iTh7PGOVonOWSNbPCldnjXyJJK0NYloqunrKaBlbCyjsJMQpGV4GOx6zvzWtxnPN9bvmEMDqSSF0HdcF+oMJf31bcjFJRqcYy1tUId1uf4RdfN1nCEcyrcHipSgT8ki+GoShA8faayD/v3PhmverG2YSNyxvWfQ5bfK9vPncOPs0wukrY39EKdy7I7CmpMCWGigy9+0zYBXAeW/4lpiHsy6FKp/WLuRkidsZm4+ohTIJ8jba9vJD1Ko6x67pV86qKmtnt889QtakunwswYAkmb5Z9oE7mG5AuSiDqRxR93uyf1fU0EiHdJwx9r/8Lnh3s+N2UvevjNatLT1yVO7BvuDbf+MvIhqnj6CH1Qrj+4kWEJCzskb/m9d0bxSR2ZL1PHbVTXGz5VgWcVMGqsIdq4yhVypl2k= ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:30:06.544 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC86XmLpBDHobQdn3NGWif/qFSOW3QeM6e79vZ/ZCb2YHk00xy4YVhc+XXrE3Spq4TALoFcEge2K0yPCQVITrzG8kYfkUQiEI2J8iTh7PGOVonOWSNbPCldnjXyJJK0NYloqunrKaBlbCyjsJMQpGV4GOx6zvzWtxnPN9bvmEMDqSSF0HdcF+oMJf31bcjFJRqcYy1tUId1uf4RdfN1nCEcyrcHipSgT8ki+GoShA8faayD/v3PhmverG2YSNyxvWfQ5bfK9vPncOPs0wukrY39EKdy7I7CmpMCWGigy9+0zYBXAeW/4lpiHsy6FKp/WLuRkidsZm4+ohTIJ8jba9vJD1Ko6x67pV86qKmtnt889QtakunwswYAkmb5Z9oE7mG5AuSiDqRxR93uyf1fU0EiHdJwx9r/8Lnh3s+N2UvevjNatLT1yVO7BvuDbf+MvIhqnj6CH1Qrj+4kWEJCzskb/m9d0bxSR2ZL1PHbVTXGz5VgWcVMGqsIdq4yhVypl2k= ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-11-20T05:30:06.589 INFO:teuthology.orchestra.run.smithi187.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC86XmLpBDHobQdn3NGWif/qFSOW3QeM6e79vZ/ZCb2YHk00xy4YVhc+XXrE3Spq4TALoFcEge2K0yPCQVITrzG8kYfkUQiEI2J8iTh7PGOVonOWSNbPCldnjXyJJK0NYloqunrKaBlbCyjsJMQpGV4GOx6zvzWtxnPN9bvmEMDqSSF0HdcF+oMJf31bcjFJRqcYy1tUId1uf4RdfN1nCEcyrcHipSgT8ki+GoShA8faayD/v3PhmverG2YSNyxvWfQ5bfK9vPncOPs0wukrY39EKdy7I7CmpMCWGigy9+0zYBXAeW/4lpiHsy6FKp/WLuRkidsZm4+ohTIJ8jba9vJD1Ko6x67pV86qKmtnt889QtakunwswYAkmb5Z9oE7mG5AuSiDqRxR93uyf1fU0EiHdJwx9r/8Lnh3s+N2UvevjNatLT1yVO7BvuDbf+MvIhqnj6CH1Qrj+4kWEJCzskb/m9d0bxSR2ZL1PHbVTXGz5VgWcVMGqsIdq4yhVypl2k= ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:30:06.602 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2021-11-20T05:30:07.444 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:07 smithi058 conmon[33537]: cluster 2021-11-20T05:30:06.322473+0000 mon.smithi058 (mon.0) 114 : cluster [DBG] mgrmap e17: smithi058.zagdlk(active, since 7s) 2021-11-20T05:30:08.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.801+0000 7fa12073c700 1 -- 172.21.15.58:0/3239952919 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1180f4b00 msgr2=0x7fa1180f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:08.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.801+0000 7fa12073c700 1 --2- 172.21.15.58:0/3239952919 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1180f4b00 0x7fa1180f4f20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fa114009fb0 tx=0x7fa114009b30).stop 2021-11-20T05:30:08.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.802+0000 7fa12073c700 1 -- 172.21.15.58:0/3239952919 shutdown_connections 2021-11-20T05:30:08.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.802+0000 7fa12073c700 1 --2- 172.21.15.58:0/3239952919 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1180f4b00 0x7fa1180f4f20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:08.805 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.802+0000 7fa12073c700 1 -- 172.21.15.58:0/3239952919 >> 172.21.15.58:0/3239952919 conn(0x7fa1180f0260 msgr2=0x7fa1180f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:08.805 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.802+0000 7fa12073c700 1 -- 172.21.15.58:0/3239952919 shutdown_connections 2021-11-20T05:30:08.805 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.802+0000 7fa12073c700 1 -- 172.21.15.58:0/3239952919 wait complete. 2021-11-20T05:30:08.805 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.802+0000 7fa12073c700 1 Processor -- start 2021-11-20T05:30:08.806 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.802+0000 7fa12073c700 1 -- start start 2021-11-20T05:30:08.806 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.802+0000 7fa12073c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1180f4b00 0x7fa118109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:08.806 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.802+0000 7fa12073c700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1180f1870 con 0x7fa1180f4b00 2021-11-20T05:30:08.806 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.803+0000 7fa11e4d8700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1180f4b00 0x7fa118109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:08.807 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.803+0000 7fa11e4d8700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1180f4b00 0x7fa118109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39860/0 (socket says 172.21.15.58:39860) 2021-11-20T05:30:08.807 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.803+0000 7fa11e4d8700 1 -- 172.21.15.58:0/3702479695 learned_addr learned my addr 172.21.15.58:0/3702479695 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:30:08.807 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.803+0000 7fa11e4d8700 1 -- 172.21.15.58:0/3702479695 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa114005040 con 0x7fa1180f4b00 2021-11-20T05:30:08.807 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.803+0000 7fa11e4d8700 1 --2- 172.21.15.58:0/3702479695 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1180f4b00 0x7fa118109420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa114013510 tx=0x7fa11400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:08.808 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.803+0000 7fa10effd700 1 -- 172.21.15.58:0/3702479695 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa114009320 con 0x7fa1180f4b00 2021-11-20T05:30:08.808 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.803+0000 7fa10effd700 1 -- 172.21.15.58:0/3702479695 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa114013f00 con 0x7fa1180f4b00 2021-11-20T05:30:08.808 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.803+0000 7fa12073c700 1 -- 172.21.15.58:0/3702479695 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1180f1a70 con 0x7fa1180f4b00 2021-11-20T05:30:08.808 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.803+0000 7fa10effd700 1 -- 172.21.15.58:0/3702479695 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa114014d90 con 0x7fa1180f4b00 2021-11-20T05:30:08.808 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.803+0000 7fa12073c700 1 -- 172.21.15.58:0/3702479695 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa118109ad0 con 0x7fa1180f4b00 2021-11-20T05:30:08.809 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.804+0000 7fa10effd700 1 -- 172.21.15.58:0/3702479695 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1140263a0 con 0x7fa1180f4b00 2021-11-20T05:30:08.809 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.804+0000 7fa12073c700 1 -- 172.21.15.58:0/3702479695 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa118059100 con 0x7fa1180f4b00 2021-11-20T05:30:08.811 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.804+0000 7fa10effd700 1 --2- 172.21.15.58:0/3702479695 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa10402ffb0 0x7fa104032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:08.811 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.804+0000 7fa10effd700 1 -- 172.21.15.58:0/3702479695 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa114041f00 con 0x7fa1180f4b00 2021-11-20T05:30:08.811 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.804+0000 7fa11dcd7700 1 --2- 172.21.15.58:0/3702479695 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa10402ffb0 0x7fa104032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:08.812 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.805+0000 7fa11dcd7700 1 --2- 172.21.15.58:0/3702479695 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa10402ffb0 0x7fa104032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa1080096a0 tx=0x7fa108006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:08.812 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.809+0000 7fa10effd700 1 -- 172.21.15.58:0/3702479695 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa114012920 con 0x7fa1180f4b00 2021-11-20T05:30:08.969 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.969+0000 7fa12073c700 1 -- 172.21.15.58:0/3702479695 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fa118045ad0 con 0x7fa1180f4b00 2021-11-20T05:30:08.972 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.972+0000 7fa10effd700 1 -- 172.21.15.58:0/3702479695 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7fa114044070 con 0x7fa1180f4b00 2021-11-20T05:30:08.974 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.974+0000 7fa12073c700 1 -- 172.21.15.58:0/3702479695 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa10402ffb0 msgr2=0x7fa104032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:08.974 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.974+0000 7fa12073c700 1 --2- 172.21.15.58:0/3702479695 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa10402ffb0 0x7fa104032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa1080096a0 tx=0x7fa108006b40).stop 2021-11-20T05:30:08.974 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.974+0000 7fa12073c700 1 -- 172.21.15.58:0/3702479695 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1180f4b00 msgr2=0x7fa118109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:08.975 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.974+0000 7fa12073c700 1 --2- 172.21.15.58:0/3702479695 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1180f4b00 0x7fa118109420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa114013510 tx=0x7fa11400b040).stop 2021-11-20T05:30:08.975 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.974+0000 7fa12073c700 1 -- 172.21.15.58:0/3702479695 shutdown_connections 2021-11-20T05:30:08.975 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.974+0000 7fa12073c700 1 --2- 172.21.15.58:0/3702479695 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa10402ffb0 0x7fa104032470 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:08.975 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.974+0000 7fa12073c700 1 --2- 172.21.15.58:0/3702479695 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1180f4b00 0x7fa118109420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:08.977 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.974+0000 7fa12073c700 1 -- 172.21.15.58:0/3702479695 >> 172.21.15.58:0/3702479695 conn(0x7fa1180f0260 msgr2=0x7fa1180f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:08.978 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.975+0000 7fa12073c700 1 -- 172.21.15.58:0/3702479695 shutdown_connections 2021-11-20T05:30:08.979 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:08.975+0000 7fa12073c700 1 -- 172.21.15.58:0/3702479695 wait complete. 2021-11-20T05:30:09.426 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi187 2021-11-20T05:30:09.426 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2021-11-20T05:30:09.427 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/etc/ceph/ceph.conf 2021-11-20T05:30:09.446 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2021-11-20T05:30:09.446 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/etc/ceph/ceph.client.admin.keyring 2021-11-20T05:30:09.504 INFO:tasks.cephadm:Adding host smithi187 to orchestrator... 2021-11-20T05:30:09.505 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph orch host add smithi187 2021-11-20T05:30:09.510 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:09 smithi058 conmon[33537]: audit 2021-11-20T05:30:08.300493+0000 mon.smithi058 (mon.0) 115 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:09.511 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:09 smithi058 conmon[33537]: audit 2021-11-20T05:30:08.829490+0000 mon.smithi058 (mon.0) 116 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:09.511 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:09 smithi058 conmon[33537]: audit 2021-11-20T05:30:08.973392+0000 mon.smithi058 (mon.0) 117 : audit [INF] from='client.? 172.21.15.58:0/3702479695' entity='client.admin' 2021-11-20T05:30:10.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:10 smithi058 conmon[33537]: cephadm 2021-11-20T05:30:08.838289+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:30:10.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:10 smithi058 conmon[33537]: ) 5 : cephadm [INF] Deploying daemon alertmanager.smithi058 on smithi058 2021-11-20T05:30:18.102 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:17 smithi058 conmon[33537]: audit 2021-11-20T05:30:16.952392+0000 mon.smithi058 (mon.0) 118 : audit [INF] 2021-11-20T05:30:18.102 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:17 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:18.102 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:17 smithi058 conmon[33537]: audit 2021-11-20T05:30:16.953433+0000 mon.smithi058 (mon.0) 119 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi058", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T05:30:18.103 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:17 smithi058 conmon[33537]: audit 2021-11-20T05:30:16.955085+0000 mon.smithi058 (mon.0) 120 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi058", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2021-11-20T05:30:18.103 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:17 smithi058 conmon[33537]: audit 2021-11-20T05:30:16.955766+0000 mon.smithi058 (mon.0) 121 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:30:18.103 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:17 smithi058 conmon[33537]: cephadm 2021-11-20T05:30:16. 2021-11-20T05:30:18.103 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:17 smithi058 conmon[33537]: 956483+0000 mgr.smithi058.zagdlk (mgr.14164) 6 : cephadm [INF] Deploying daemon crash.smithi058 on smithi058 2021-11-20T05:30:20.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:20 smithi058 conmon[33537]: audit 2021-11-20T05:30:19.147746+0000 2021-11-20T05:30:20.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:20 smithi058 conmon[33537]: mon.smithi058 (mon.0) 122 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:20.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:20 smithi058 conmon[33537]: audit 2021-11-20T05:30:19.228081+0000 mon.smithi058 (mon.0) 123 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:20.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:20 smithi058 conmon[33537]: audit 2021-11-20T05:30:19.229990+0000 mon.smithi058 (mon.0) 124 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:20.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:20 smithi058 conmon[33537]: audit 2021-11-20T05:30:19.230400+0000 mon.smithi058 (mon.0) 125 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-11-20T05:30:20.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:20 smithi058 conmon[33537]: audit 2021-11-20T05:30:19.230904+0000 mgr.smithi058.zagdlk (mgr.14164) 7 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-11-20T05:30:20.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:20 smithi058 conmon[33537]: audit 2021-11-20T05:30:19.233328+0000 mon.smithi058 (mon.0) 126 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:20.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:20 smithi058 conmon[33537]: cephadm 2021-11-20T05:30:19.237369+0000 mgr.smithi058.zagdlk (mgr.14164) 8 : cephadm [INF] Deploying daemon grafana.smithi058 on smithi058 2021-11-20T05:30:21.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:21 smithi058 conmon[33537]: cluster 2021-11-20T05:30:20.171628+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:30:21.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:21 smithi058 conmon[33537]: mgr.14164) 9 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:23.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:23 smithi058 conmon[33537]: cluster 2021-11-20T05:30:22.171861+0000 mgr.smithi058.zagdlk 2021-11-20T05:30:23.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:23 smithi058 conmon[33537]: (mgr.14164) 10 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:24.990 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.987+0000 7faf8e7e1700 1 -- 172.21.15.187:0/2357694485 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf880f47c0 msgr2=0x7faf880f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:24.990 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.987+0000 7faf8e7e1700 1 --2- 172.21.15.187:0/2357694485 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf880f47c0 0x7faf880f4be0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7faf78009fb0 tx=0x7faf78009b30).stop 2021-11-20T05:30:24.991 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.987+0000 7faf8e7e1700 1 -- 172.21.15.187:0/2357694485 shutdown_connections 2021-11-20T05:30:24.991 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.987+0000 7faf8e7e1700 1 --2- 172.21.15.187:0/2357694485 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf880f47c0 0x7faf880f4be0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:24.991 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.987+0000 7faf8e7e1700 1 -- 172.21.15.187:0/2357694485 >> 172.21.15.187:0/2357694485 conn(0x7faf880f02e0 msgr2=0x7faf880f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:24.991 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.987+0000 7faf8e7e1700 1 -- 172.21.15.187:0/2357694485 shutdown_connections 2021-11-20T05:30:24.991 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.987+0000 7faf8e7e1700 1 -- 172.21.15.187:0/2357694485 wait complete. 2021-11-20T05:30:24.992 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.987+0000 7faf8e7e1700 1 Processor -- start 2021-11-20T05:30:24.992 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.988+0000 7faf8e7e1700 1 -- start start 2021-11-20T05:30:24.992 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.988+0000 7faf8e7e1700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf880f47c0 0x7faf88106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:24.992 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.988+0000 7faf8e7e1700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf88106bf0 con 0x7faf880f47c0 2021-11-20T05:30:24.993 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.988+0000 7faf87fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf880f47c0 0x7faf88106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:24.993 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.988+0000 7faf87fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf880f47c0 0x7faf88106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.187:46044/0 (socket says 172.21.15.187:46044) 2021-11-20T05:30:24.993 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.988+0000 7faf87fff700 1 -- 172.21.15.187:0/953157363 learned_addr learned my addr 172.21.15.187:0/953157363 (peer_addr_for_me v2:172.21.15.187:0/0) 2021-11-20T05:30:24.993 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.988+0000 7faf87fff700 1 -- 172.21.15.187:0/953157363 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf78005040 con 0x7faf880f47c0 2021-11-20T05:30:24.994 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.988+0000 7faf87fff700 1 --2- 172.21.15.187:0/953157363 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf880f47c0 0x7faf88106700 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7faf78013510 tx=0x7faf7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:24.994 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.989+0000 7faf84ff9700 1 -- 172.21.15.187:0/953157363 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf78009320 con 0x7faf880f47c0 2021-11-20T05:30:24.994 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.989+0000 7faf8e7e1700 1 -- 172.21.15.187:0/953157363 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf88106df0 con 0x7faf880f47c0 2021-11-20T05:30:24.994 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.989+0000 7faf84ff9700 1 -- 172.21.15.187:0/953157363 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf78013f00 con 0x7faf880f47c0 2021-11-20T05:30:24.995 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.989+0000 7faf84ff9700 1 -- 172.21.15.187:0/953157363 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf78014d90 con 0x7faf880f47c0 2021-11-20T05:30:24.995 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.989+0000 7faf8e7e1700 1 -- 172.21.15.187:0/953157363 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf88107280 con 0x7faf880f47c0 2021-11-20T05:30:24.995 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.990+0000 7faf84ff9700 1 -- 172.21.15.187:0/953157363 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf780095f0 con 0x7faf880f47c0 2021-11-20T05:30:24.996 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.990+0000 7faf8e7e1700 1 -- 172.21.15.187:0/953157363 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf88045ad0 con 0x7faf880f47c0 2021-11-20T05:30:24.996 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.990+0000 7faf84ff9700 1 --2- 172.21.15.187:0/953157363 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7faf7002ff60 0x7faf70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:24.996 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.990+0000 7faf84ff9700 1 -- 172.21.15.187:0/953157363 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf78040fe0 con 0x7faf880f47c0 2021-11-20T05:30:24.997 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.990+0000 7faf877fe700 1 --2- 172.21.15.187:0/953157363 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7faf7002ff60 0x7faf70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:24.997 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.991+0000 7faf877fe700 1 --2- 172.21.15.187:0/953157363 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7faf7002ff60 0x7faf70032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7faf7c0096a0 tx=0x7faf7c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:24.998 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:24.998+0000 7faf84ff9700 1 -- 172.21.15.187:0/953157363 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf7801d7f0 con 0x7faf880f47c0 2021-11-20T05:30:25.168 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:25.167+0000 7faf8e7e1700 1 -- 172.21.15.187:0/953157363 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi187", "target": ["mon-mgr", ""]}) v1 -- 0x7faf880f14c0 con 0x7faf7002ff60 2021-11-20T05:30:25.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:25 smithi058 conmon[33537]: cluster 2021-11-20T05:30:24.172096+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:30:25.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:25 smithi058 conmon[33537]: 14164) 11 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:25.881 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:25.880+0000 7faf84ff9700 1 -- 172.21.15.187:0/953157363 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7faf880f14c0 con 0x7faf7002ff60 2021-11-20T05:30:25.882 INFO:teuthology.orchestra.run.smithi187.stdout:Added host 'smithi187' 2021-11-20T05:30:25.883 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:25.882+0000 7faf8e7e1700 1 -- 172.21.15.187:0/953157363 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7faf7002ff60 msgr2=0x7faf70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:25.883 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:25.882+0000 7faf8e7e1700 1 --2- 172.21.15.187:0/953157363 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7faf7002ff60 0x7faf70032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7faf7c0096a0 tx=0x7faf7c006b40).stop 2021-11-20T05:30:25.883 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:25.882+0000 7faf8e7e1700 1 -- 172.21.15.187:0/953157363 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf880f47c0 msgr2=0x7faf88106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:25.884 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:25.882+0000 7faf8e7e1700 1 --2- 172.21.15.187:0/953157363 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf880f47c0 0x7faf88106700 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7faf78013510 tx=0x7faf7800b040).stop 2021-11-20T05:30:25.884 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:25.882+0000 7faf8e7e1700 1 -- 172.21.15.187:0/953157363 shutdown_connections 2021-11-20T05:30:25.884 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:25.882+0000 7faf8e7e1700 1 --2- 172.21.15.187:0/953157363 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7faf7002ff60 0x7faf70032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:25.884 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:25.882+0000 7faf8e7e1700 1 --2- 172.21.15.187:0/953157363 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf880f47c0 0x7faf88106700 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:25.885 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:25.882+0000 7faf8e7e1700 1 -- 172.21.15.187:0/953157363 >> 172.21.15.187:0/953157363 conn(0x7faf880f02e0 msgr2=0x7faf880f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:25.885 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:25.882+0000 7faf8e7e1700 1 -- 172.21.15.187:0/953157363 shutdown_connections 2021-11-20T05:30:25.885 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:25.882+0000 7faf8e7e1700 1 -- 172.21.15.187:0/953157363 wait complete. 2021-11-20T05:30:26.296 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph orch host ls --format=json 2021-11-20T05:30:26.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:26 smithi058 conmon[33537]: audit 2021-11-20T05:30:25.169953+0000 mgr.smithi058.zagdlk 2021-11-20T05:30:26.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:26 smithi058 conmon[33537]: (mgr.14164) 12 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi187", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:30:26.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:26 smithi058 conmon[33537]: audit 2021-11-20T05:30:25.880978+0000 mon.smithi058 2021-11-20T05:30:26.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:26 smithi058 conmon[33537]: (mon.0) 127 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:27.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:27 smithi058 conmon[33537]: cephadm 2021-11-20T05:30:25.881490+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:30:27.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:27 smithi058 conmon[33537]: mgr.14164) 13 : cephadm [INF] Added host smithi187 2021-11-20T05:30:27.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:27 smithi058 conmon[33537]: cluster 2021-11-20T05:30:26.172309+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:30:27.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:27 smithi058 conmon[33537]: 14164) 14 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:27.949 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.945+0000 7fb08d0ef700 1 -- 172.21.15.187:0/4217668307 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb0880f6510 msgr2=0x7fb0880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:27.949 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.945+0000 7fb08d0ef700 1 --2- 172.21.15.187:0/4217668307 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb0880f6510 0x7fb0880f6930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fb078009fb0 tx=0x7fb078009b30).stop 2021-11-20T05:30:27.950 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.945+0000 7fb08d0ef700 1 -- 172.21.15.187:0/4217668307 shutdown_connections 2021-11-20T05:30:27.950 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.945+0000 7fb08d0ef700 1 --2- 172.21.15.187:0/4217668307 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb0880f6510 0x7fb0880f6930 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:27.950 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.945+0000 7fb08d0ef700 1 -- 172.21.15.187:0/4217668307 >> 172.21.15.187:0/4217668307 conn(0x7fb0880f1c70 msgr2=0x7fb0880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:27.950 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.945+0000 7fb08d0ef700 1 -- 172.21.15.187:0/4217668307 shutdown_connections 2021-11-20T05:30:27.951 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.945+0000 7fb08d0ef700 1 -- 172.21.15.187:0/4217668307 wait complete. 2021-11-20T05:30:27.951 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.946+0000 7fb08d0ef700 1 Processor -- start 2021-11-20T05:30:27.951 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.946+0000 7fb08d0ef700 1 -- start start 2021-11-20T05:30:27.951 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.946+0000 7fb08d0ef700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb0880f6510 0x7fb088106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:27.951 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.946+0000 7fb08d0ef700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb088106c20 con 0x7fb0880f6510 2021-11-20T05:30:27.952 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.947+0000 7fb086d9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb0880f6510 0x7fb088106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:27.952 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.947+0000 7fb086d9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb0880f6510 0x7fb088106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.187:46050/0 (socket says 172.21.15.187:46050) 2021-11-20T05:30:27.952 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.947+0000 7fb086d9d700 1 -- 172.21.15.187:0/3528206352 learned_addr learned my addr 172.21.15.187:0/3528206352 (peer_addr_for_me v2:172.21.15.187:0/0) 2021-11-20T05:30:27.952 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.947+0000 7fb086d9d700 1 -- 172.21.15.187:0/3528206352 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb078005040 con 0x7fb0880f6510 2021-11-20T05:30:27.953 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.947+0000 7fb086d9d700 1 --2- 172.21.15.187:0/3528206352 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb0880f6510 0x7fb088106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fb078014000 tx=0x7fb07800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:27.953 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.947+0000 7fb06f7fe700 1 -- 172.21.15.187:0/3528206352 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0780095f0 con 0x7fb0880f6510 2021-11-20T05:30:27.953 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.947+0000 7fb06f7fe700 1 -- 172.21.15.187:0/3528206352 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb078007e70 con 0x7fb0880f6510 2021-11-20T05:30:27.953 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.947+0000 7fb08d0ef700 1 -- 172.21.15.187:0/3528206352 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb088106e20 con 0x7fb0880f6510 2021-11-20T05:30:27.953 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.947+0000 7fb06f7fe700 1 -- 172.21.15.187:0/3528206352 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb078014ea0 con 0x7fb0880f6510 2021-11-20T05:30:27.954 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.947+0000 7fb08d0ef700 1 -- 172.21.15.187:0/3528206352 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0881072b0 con 0x7fb0880f6510 2021-11-20T05:30:27.955 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.948+0000 7fb06f7fe700 1 -- 172.21.15.187:0/3528206352 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb078025070 con 0x7fb0880f6510 2021-11-20T05:30:27.955 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.948+0000 7fb06f7fe700 1 --2- 172.21.15.187:0/3528206352 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb07002ffb0 0x7fb070032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:27.955 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.948+0000 7fb08d0ef700 1 -- 172.21.15.187:0/3528206352 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb088045ad0 con 0x7fb0880f6510 2021-11-20T05:30:27.956 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.948+0000 7fb06f7fe700 1 -- 172.21.15.187:0/3528206352 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb078040c30 con 0x7fb0880f6510 2021-11-20T05:30:27.956 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.948+0000 7fb08659c700 1 --2- 172.21.15.187:0/3528206352 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb07002ffb0 0x7fb070032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:27.956 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.949+0000 7fb08659c700 1 --2- 172.21.15.187:0/3528206352 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb07002ffb0 0x7fb070032470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fb07c0096a0 tx=0x7fb07c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:27.957 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:27.953+0000 7fb06f7fe700 1 -- 172.21.15.187:0/3528206352 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb078026350 con 0x7fb0880f6510 2021-11-20T05:30:28.126 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:28.124+0000 7fb08d0ef700 1 -- 172.21.15.187:0/3528206352 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb0880f2e70 con 0x7fb07002ffb0 2021-11-20T05:30:28.126 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:28.125+0000 7fb06f7fe700 1 -- 172.21.15.187:0/3528206352 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7fb0880f2e70 con 0x7fb07002ffb0 2021-11-20T05:30:28.126 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:30:28.127 INFO:teuthology.orchestra.run.smithi187.stdout:[{"addr": "172.21.15.58", "hostname": "smithi058", "labels": [], "status": ""}, {"addr": "smithi187", "hostname": "smithi187", "labels": [], "status": ""}] 2021-11-20T05:30:28.128 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:28.127+0000 7fb08d0ef700 1 -- 172.21.15.187:0/3528206352 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb07002ffb0 msgr2=0x7fb070032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:28.128 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:28.127+0000 7fb08d0ef700 1 --2- 172.21.15.187:0/3528206352 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb07002ffb0 0x7fb070032470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fb07c0096a0 tx=0x7fb07c006b40).stop 2021-11-20T05:30:28.128 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:28.127+0000 7fb08d0ef700 1 -- 172.21.15.187:0/3528206352 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb0880f6510 msgr2=0x7fb088106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:28.128 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:28.127+0000 7fb08d0ef700 1 --2- 172.21.15.187:0/3528206352 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb0880f6510 0x7fb088106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fb078014000 tx=0x7fb07800b040).stop 2021-11-20T05:30:28.129 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:28.127+0000 7fb08d0ef700 1 -- 172.21.15.187:0/3528206352 shutdown_connections 2021-11-20T05:30:28.129 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:28.127+0000 7fb08d0ef700 1 --2- 172.21.15.187:0/3528206352 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb07002ffb0 0x7fb070032470 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:28.129 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:28.127+0000 7fb08d0ef700 1 --2- 172.21.15.187:0/3528206352 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb0880f6510 0x7fb088106730 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:28.129 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:28.127+0000 7fb08d0ef700 1 -- 172.21.15.187:0/3528206352 >> 172.21.15.187:0/3528206352 conn(0x7fb0880f1c70 msgr2=0x7fb0880f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:28.130 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:28.127+0000 7fb08d0ef700 1 -- 172.21.15.187:0/3528206352 shutdown_connections 2021-11-20T05:30:28.130 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:28.127+0000 7fb08d0ef700 1 -- 172.21.15.187:0/3528206352 wait complete. 2021-11-20T05:30:28.623 INFO:tasks.cephadm:Setting crush tunables to default 2021-11-20T05:30:28.624 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph osd crush tunables default 2021-11-20T05:30:29.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:29 smithi058 conmon[33537]: audit 2021-11-20T05:30:28.126633+0000 mgr.smithi058.zagdlk (mgr.14164) 15 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T05:30:29.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:29 smithi058 conmon[33537]: cluster 2021-11-20T05:30:28.172538+0000 mgr.smithi058.zagdlk (mgr.14164) 16 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:30.543 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.542+0000 7fe5acc10700 1 -- 172.21.15.58:0/1419560495 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe5a80f62a0 msgr2=0x7fe5a80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:30.544 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.542+0000 7fe5acc10700 1 --2- 172.21.15.58:0/1419560495 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe5a80f62a0 0x7fe5a80f66c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe598009960 tx=0x7fe598004d10).stop 2021-11-20T05:30:30.544 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.542+0000 7fe5acc10700 1 -- 172.21.15.58:0/1419560495 shutdown_connections 2021-11-20T05:30:30.544 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.542+0000 7fe5acc10700 1 --2- 172.21.15.58:0/1419560495 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe5a80f62a0 0x7fe5a80f66c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:30.545 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.542+0000 7fe5acc10700 1 -- 172.21.15.58:0/1419560495 >> 172.21.15.58:0/1419560495 conn(0x7fe5a80f1aa0 msgr2=0x7fe5a80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:30.545 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.542+0000 7fe5acc10700 1 -- 172.21.15.58:0/1419560495 shutdown_connections 2021-11-20T05:30:30.545 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.542+0000 7fe5acc10700 1 -- 172.21.15.58:0/1419560495 wait complete. 2021-11-20T05:30:30.545 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.543+0000 7fe5acc10700 1 Processor -- start 2021-11-20T05:30:30.546 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.543+0000 7fe5acc10700 1 -- start start 2021-11-20T05:30:30.546 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.543+0000 7fe5acc10700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe5a80f62a0 0x7fe5a806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:30.546 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.543+0000 7fe5acc10700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a8069e20 con 0x7fe5a80f62a0 2021-11-20T05:30:30.546 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.543+0000 7fe5a77fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe5a80f62a0 0x7fe5a806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:30.547 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.543+0000 7fe5a77fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe5a80f62a0 0x7fe5a806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39902/0 (socket says 172.21.15.58:39902) 2021-11-20T05:30:30.547 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.543+0000 7fe5a77fe700 1 -- 172.21.15.58:0/905785514 learned_addr learned my addr 172.21.15.58:0/905785514 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:30:30.547 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.543+0000 7fe5a77fe700 1 -- 172.21.15.58:0/905785514 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe598009640 con 0x7fe5a80f62a0 2021-11-20T05:30:30.547 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.544+0000 7fe5a77fe700 1 --2- 172.21.15.58:0/905785514 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe5a80f62a0 0x7fe5a806b9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fe598007b20 tx=0x7fe598004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:30.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.544+0000 7fe58ffff700 1 -- 172.21.15.58:0/905785514 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe598018070 con 0x7fe5a80f62a0 2021-11-20T05:30:30.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.544+0000 7fe5acc10700 1 -- 172.21.15.58:0/905785514 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a8069fc0 con 0x7fe5a80f62a0 2021-11-20T05:30:30.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.544+0000 7fe58ffff700 1 -- 172.21.15.58:0/905785514 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe598007e60 con 0x7fe5a80f62a0 2021-11-20T05:30:30.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.544+0000 7fe58ffff700 1 -- 172.21.15.58:0/905785514 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe59801d420 con 0x7fe5a80f62a0 2021-11-20T05:30:30.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.544+0000 7fe5acc10700 1 -- 172.21.15.58:0/905785514 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a806a450 con 0x7fe5a80f62a0 2021-11-20T05:30:30.550 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.545+0000 7fe58ffff700 1 -- 172.21.15.58:0/905785514 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe598013630 con 0x7fe5a80f62a0 2021-11-20T05:30:30.550 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.545+0000 7fe58ffff700 1 --2- 172.21.15.58:0/905785514 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe59002ff60 0x7fe590032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:30.550 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.545+0000 7fe58ffff700 1 -- 172.21.15.58:0/905785514 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe5980416f0 con 0x7fe5a80f62a0 2021-11-20T05:30:30.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.545+0000 7fe5a6ffd700 1 --2- 172.21.15.58:0/905785514 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe59002ff60 0x7fe590032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:30.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.545+0000 7fe5acc10700 1 -- 172.21.15.58:0/905785514 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a8059100 con 0x7fe5a80f62a0 2021-11-20T05:30:30.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.545+0000 7fe5a6ffd700 1 --2- 172.21.15.58:0/905785514 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe59002ff60 0x7fe590032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fe59c0096a0 tx=0x7fe59c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:30.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.550+0000 7fe58ffff700 1 -- 172.21.15.58:0/905785514 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe5980263e0 con 0x7fe5a80f62a0 2021-11-20T05:30:30.722 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:30.721+0000 7fe5acc10700 1 -- 172.21.15.58:0/905785514 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe5a806ab50 con 0x7fe5a80f62a0 2021-11-20T05:30:31.138 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:31.137+0000 7fe58ffff700 1 -- 172.21.15.58:0/905785514 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe598011320 con 0x7fe5a80f62a0 2021-11-20T05:30:31.139 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:31.139+0000 7fe5acc10700 1 -- 172.21.15.58:0/905785514 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe59002ff60 msgr2=0x7fe590032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:31.139 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:31.139+0000 7fe5acc10700 1 --2- 172.21.15.58:0/905785514 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe59002ff60 0x7fe590032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fe59c0096a0 tx=0x7fe59c006b40).stop 2021-11-20T05:30:31.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:31.139+0000 7fe5acc10700 1 -- 172.21.15.58:0/905785514 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe5a80f62a0 msgr2=0x7fe5a806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:31.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:31.139+0000 7fe5acc10700 1 --2- 172.21.15.58:0/905785514 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe5a80f62a0 0x7fe5a806b9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fe598007b20 tx=0x7fe598004d10).stop 2021-11-20T05:30:31.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:31.139+0000 7fe5acc10700 1 -- 172.21.15.58:0/905785514 shutdown_connections 2021-11-20T05:30:31.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:31.139+0000 7fe5acc10700 1 --2- 172.21.15.58:0/905785514 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe59002ff60 0x7fe590032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:31.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:31.139+0000 7fe5acc10700 1 --2- 172.21.15.58:0/905785514 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe5a80f62a0 0x7fe5a806b9a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:31.141 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:31.139+0000 7fe5acc10700 1 -- 172.21.15.58:0/905785514 >> 172.21.15.58:0/905785514 conn(0x7fe5a80f1aa0 msgr2=0x7fe5a80f25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:31.141 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:31.140+0000 7fe5acc10700 1 -- 172.21.15.58:0/905785514 shutdown_connections 2021-11-20T05:30:31.141 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:30:31.140+0000 7fe5acc10700 1 -- 172.21.15.58:0/905785514 wait complete. 2021-11-20T05:30:31.142 INFO:teuthology.orchestra.run.smithi058.stderr:adjusted tunables profile to default 2021-11-20T05:30:31.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:31 smithi058 conmon[33537]: audit 2021-11-20T05:30:30 2021-11-20T05:30:31.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:31 smithi058 conmon[33537]: .135592+0000 mon.smithi058 (mon.0) 128 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:31.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:31 smithi058 conmon[33537]: cephadm 2021-11-20T05:30:30.137247+0000 mgr.smithi058.zagdlk (mgr.14164) 17 : cephadm [INF] Deploying daemon node-exporter.smithi058 on smithi058 2021-11-20T05:30:31.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:31 smithi058 conmon[33537]: cluster 2021-11-20T05:30:30.172777+0000 mgr.smithi058.zagdlk (mgr.14164) 18 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:31.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:31 smithi058 conmon[33537]: audit 2021-11-20T05:30:30.722927+0000 mon.smithi058 (mon.0) 129 : audit [INF] from='client.? 172.21.15.58:0/905785514' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2021-11-20T05:30:31.607 INFO:tasks.cephadm:Adding mon.smithi058 on smithi058 2021-11-20T05:30:31.608 INFO:tasks.cephadm:Adding mon.smithi187 on smithi187 2021-11-20T05:30:31.608 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph orch apply mon '2;smithi058:172.21.15.58=smithi058;smithi187:172.21.15.187=smithi187' 2021-11-20T05:30:32.455 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:32 smithi058 conmon[33537]: audit 2021-11-20T05:30:31.138635+0000 mon.smithi058 2021-11-20T05:30:32.456 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:32 smithi058 conmon[33537]: (mon.0) 130 : audit [INF] from='client.? 172.21.15.58:0/905785514' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2021-11-20T05:30:32.456 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:32 smithi058 conmon[33537]: cluster 2021-11-20T05:30:31.138687+0000 mon.smithi058 (mon.0) 131 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2021-11-20T05:30:33.262 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:33 smithi058 conmon[33537]: cluster 2021-11-20T05:30:32.173119+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:30:33.262 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:33 smithi058 conmon[33537]: ) 19 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:33.347 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.345+0000 7f96721d6700 1 -- 172.21.15.187:0/570688847 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f966c0f6510 msgr2=0x7f966c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:33.347 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.345+0000 7f96721d6700 1 --2- 172.21.15.187:0/570688847 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f966c0f6510 0x7f966c0f6930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f965c009fb0 tx=0x7f965c009b30).stop 2021-11-20T05:30:33.348 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.345+0000 7f96721d6700 1 -- 172.21.15.187:0/570688847 shutdown_connections 2021-11-20T05:30:33.348 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.345+0000 7f96721d6700 1 --2- 172.21.15.187:0/570688847 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f966c0f6510 0x7f966c0f6930 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:33.348 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.345+0000 7f96721d6700 1 -- 172.21.15.187:0/570688847 >> 172.21.15.187:0/570688847 conn(0x7f966c0f1c70 msgr2=0x7f966c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:33.348 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.345+0000 7f96721d6700 1 -- 172.21.15.187:0/570688847 shutdown_connections 2021-11-20T05:30:33.349 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.345+0000 7f96721d6700 1 -- 172.21.15.187:0/570688847 wait complete. 2021-11-20T05:30:33.349 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.345+0000 7f96721d6700 1 Processor -- start 2021-11-20T05:30:33.349 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.346+0000 7f96721d6700 1 -- start start 2021-11-20T05:30:33.349 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.346+0000 7f96721d6700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f966c0f6510 0x7f966c106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:33.349 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.346+0000 7f96721d6700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c106bf0 con 0x7f966c0f6510 2021-11-20T05:30:33.350 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.346+0000 7f966b7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f966c0f6510 0x7f966c106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:33.350 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.346+0000 7f966b7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f966c0f6510 0x7f966c106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.187:46056/0 (socket says 172.21.15.187:46056) 2021-11-20T05:30:33.350 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.346+0000 7f966b7fe700 1 -- 172.21.15.187:0/1287065076 learned_addr learned my addr 172.21.15.187:0/1287065076 (peer_addr_for_me v2:172.21.15.187:0/0) 2021-11-20T05:30:33.350 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.346+0000 7f966b7fe700 1 -- 172.21.15.187:0/1287065076 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c005040 con 0x7f966c0f6510 2021-11-20T05:30:33.351 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.346+0000 7f966b7fe700 1 --2- 172.21.15.187:0/1287065076 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f966c0f6510 0x7f966c106700 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f965c013510 tx=0x7f965c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:33.351 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.346+0000 7f9653fff700 1 -- 172.21.15.187:0/1287065076 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f965c009320 con 0x7f966c0f6510 2021-11-20T05:30:33.351 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.346+0000 7f9653fff700 1 -- 172.21.15.187:0/1287065076 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f965c013f00 con 0x7f966c0f6510 2021-11-20T05:30:33.351 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.347+0000 7f9653fff700 1 -- 172.21.15.187:0/1287065076 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f965c014d90 con 0x7f966c0f6510 2021-11-20T05:30:33.352 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.347+0000 7f96721d6700 1 -- 172.21.15.187:0/1287065076 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f966c106df0 con 0x7f966c0f6510 2021-11-20T05:30:33.352 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.347+0000 7f96721d6700 1 -- 172.21.15.187:0/1287065076 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f966c107280 con 0x7f966c0f6510 2021-11-20T05:30:33.353 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.347+0000 7f9653fff700 1 -- 172.21.15.187:0/1287065076 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f965c0095f0 con 0x7f966c0f6510 2021-11-20T05:30:33.353 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.347+0000 7f96721d6700 1 -- 172.21.15.187:0/1287065076 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f966c045ad0 con 0x7f966c0f6510 2021-11-20T05:30:33.354 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.347+0000 7f9653fff700 1 --2- 172.21.15.187:0/1287065076 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f965402ffb0 0x7f9654032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:33.354 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.348+0000 7f9653fff700 1 -- 172.21.15.187:0/1287065076 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f965c040ff0 con 0x7f966c0f6510 2021-11-20T05:30:33.354 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.348+0000 7f966affd700 1 --2- 172.21.15.187:0/1287065076 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f965402ffb0 0x7f9654032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:33.354 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.348+0000 7f966affd700 1 --2- 172.21.15.187:0/1287065076 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f965402ffb0 0x7f9654032470 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f96600096a0 tx=0x7f9660006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:33.355 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.353+0000 7f9653fff700 1 -- 172.21.15.187:0/1287065076 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f965c06d2b0 con 0x7f966c0f6510 2021-11-20T05:30:33.525 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.525+0000 7f96721d6700 1 -- 172.21.15.187:0/1287065076 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi058:172.21.15.58=smithi058;smithi187:172.21.15.187=smithi187", "target": ["mon-mgr", ""]}) v1 -- 0x7f966c0f2ed0 con 0x7f965402ffb0 2021-11-20T05:30:33.531 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.532+0000 7f9653fff700 1 -- 172.21.15.187:0/1287065076 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f966c0f2ed0 con 0x7f965402ffb0 2021-11-20T05:30:33.531 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled mon update... 2021-11-20T05:30:33.532 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.532+0000 7f96721d6700 1 -- 172.21.15.187:0/1287065076 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f965402ffb0 msgr2=0x7f9654032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:33.533 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.532+0000 7f96721d6700 1 --2- 172.21.15.187:0/1287065076 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f965402ffb0 0x7f9654032470 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f96600096a0 tx=0x7f9660006b40).stop 2021-11-20T05:30:33.533 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.532+0000 7f96721d6700 1 -- 172.21.15.187:0/1287065076 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f966c0f6510 msgr2=0x7f966c106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:33.533 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.532+0000 7f96721d6700 1 --2- 172.21.15.187:0/1287065076 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f966c0f6510 0x7f966c106700 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f965c013510 tx=0x7f965c00b040).stop 2021-11-20T05:30:33.534 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.532+0000 7f96721d6700 1 -- 172.21.15.187:0/1287065076 shutdown_connections 2021-11-20T05:30:33.534 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.532+0000 7f96721d6700 1 --2- 172.21.15.187:0/1287065076 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f965402ffb0 0x7f9654032470 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:33.535 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.532+0000 7f96721d6700 1 --2- 172.21.15.187:0/1287065076 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f966c0f6510 0x7f966c106700 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:33.535 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.532+0000 7f96721d6700 1 -- 172.21.15.187:0/1287065076 >> 172.21.15.187:0/1287065076 conn(0x7f966c0f1c70 msgr2=0x7f966c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:33.536 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.532+0000 7f96721d6700 1 -- 172.21.15.187:0/1287065076 shutdown_connections 2021-11-20T05:30:33.536 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:33.532+0000 7f96721d6700 1 -- 172.21.15.187:0/1287065076 wait complete. 2021-11-20T05:30:34.054 DEBUG:teuthology.orchestra.run.smithi187:mon.smithi187> sudo journalctl -f -n 0 -u ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mon.smithi187.service 2021-11-20T05:30:34.057 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T05:30:34.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T05:30:34.083 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:-- Logs begin at Sat 2021-11-20 05:17:59 UTC. -- 2021-11-20T05:30:34.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:34 smithi058 conmon[33537]: audit 2021-11-20T05:30:33.425247+0000 mon.smithi058 (mon.0) 2021-11-20T05:30:34.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:34 smithi058 conmon[33537]: 132 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:34.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:34 smithi058 conmon[33537]: cephadm 2021-11-20T05:30:33.432388+0000 mgr.smithi058.zagdlk (mgr.14164) 20 : cephadm [INF] Deploying daemon prometheus.smithi058 on smithi058 2021-11-20T05:30:34.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:34 smithi058 conmon[33537]: audit 2021-11-20T05:30:33.528444+0000 mgr.smithi058.zagdlk (mgr.14164) 21 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi058:172.21.15.58=smithi058;smithi187:172.21.15.187=smithi187", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:30:34.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:34 smithi058 conmon[33537]: cephadm 2021-11-20T05:30:33.529774+0000 mgr.smithi058.zagdlk (mgr.14164) 22 : cephadm [INF] Saving service mon spec with placement smithi058:172.21.15.58=smithi058;smithi187:172.21.15.187=smithi187;count:2 2021-11-20T05:30:34.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:34 smithi058 conmon[33537]: audit 2021-11-20T05:30:33.532080+0000 mon.smithi058 (mon.0) 133 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:34.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:34 smithi058 conmon[33537]: cluster 2021-11-20T05:30:34.173348+0000 mgr.smithi058.zagdlk (mgr.14164) 23 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:35.669 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.668+0000 7fed8e8e7700 1 -- 172.21.15.187:0/4222458789 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fed880f51a0 msgr2=0x7fed880f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:35.669 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.668+0000 7fed8e8e7700 1 --2- 172.21.15.187:0/4222458789 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fed880f51a0 0x7fed880f55c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fed84009fb0 tx=0x7fed84009b30).stop 2021-11-20T05:30:35.670 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.668+0000 7fed8e8e7700 1 -- 172.21.15.187:0/4222458789 shutdown_connections 2021-11-20T05:30:35.670 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.668+0000 7fed8e8e7700 1 --2- 172.21.15.187:0/4222458789 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fed880f51a0 0x7fed880f55c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:35.670 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.668+0000 7fed8e8e7700 1 -- 172.21.15.187:0/4222458789 >> 172.21.15.187:0/4222458789 conn(0x7fed880f09a0 msgr2=0x7fed880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:35.670 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.668+0000 7fed8e8e7700 1 -- 172.21.15.187:0/4222458789 shutdown_connections 2021-11-20T05:30:35.671 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.669+0000 7fed8e8e7700 1 -- 172.21.15.187:0/4222458789 wait complete. 2021-11-20T05:30:35.671 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.669+0000 7fed8e8e7700 1 Processor -- start 2021-11-20T05:30:35.671 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.669+0000 7fed8e8e7700 1 -- start start 2021-11-20T05:30:35.671 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.669+0000 7fed8e8e7700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fed880601f0 0x7fed8810b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:35.672 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.669+0000 7fed8e8e7700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed88107140 con 0x7fed880601f0 2021-11-20T05:30:35.672 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.670+0000 7fed8d8e5700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fed880601f0 0x7fed8810b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:35.672 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.670+0000 7fed8d8e5700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fed880601f0 0x7fed8810b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.187:46062/0 (socket says 172.21.15.187:46062) 2021-11-20T05:30:35.672 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.670+0000 7fed8d8e5700 1 -- 172.21.15.187:0/3115523871 learned_addr learned my addr 172.21.15.187:0/3115523871 (peer_addr_for_me v2:172.21.15.187:0/0) 2021-11-20T05:30:35.672 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.670+0000 7fed8d8e5700 1 -- 172.21.15.187:0/3115523871 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed84005040 con 0x7fed880601f0 2021-11-20T05:30:35.673 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.670+0000 7fed8d8e5700 1 --2- 172.21.15.187:0/3115523871 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fed880601f0 0x7fed8810b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fed84009f80 tx=0x7fed84009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:35.673 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.670+0000 7fed7e7fc700 1 -- 172.21.15.187:0/3115523871 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed840136c0 con 0x7fed880601f0 2021-11-20T05:30:35.674 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.670+0000 7fed7e7fc700 1 -- 172.21.15.187:0/3115523871 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed840095f0 con 0x7fed880601f0 2021-11-20T05:30:35.674 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.670+0000 7fed8e8e7700 1 -- 172.21.15.187:0/3115523871 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed8810bc40 con 0x7fed880601f0 2021-11-20T05:30:35.675 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.670+0000 7fed7e7fc700 1 -- 172.21.15.187:0/3115523871 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed84014420 con 0x7fed880601f0 2021-11-20T05:30:35.675 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.670+0000 7fed8e8e7700 1 -- 172.21.15.187:0/3115523871 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8810bf90 con 0x7fed880601f0 2021-11-20T05:30:35.675 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.671+0000 7fed7e7fc700 1 -- 172.21.15.187:0/3115523871 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed84025060 con 0x7fed880601f0 2021-11-20T05:30:35.675 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.671+0000 7fed7e7fc700 1 --2- 172.21.15.187:0/3115523871 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fed7402ff60 0x7fed74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:35.675 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.671+0000 7fed8e8e7700 1 -- 172.21.15.187:0/3115523871 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed88045ad0 con 0x7fed880601f0 2021-11-20T05:30:35.676 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.671+0000 7fed7e7fc700 1 -- 172.21.15.187:0/3115523871 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed840410f0 con 0x7fed880601f0 2021-11-20T05:30:35.676 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.671+0000 7fed8d0e4700 1 --2- 172.21.15.187:0/3115523871 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fed7402ff60 0x7fed74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:35.676 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.672+0000 7fed8d0e4700 1 --2- 172.21.15.187:0/3115523871 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fed7402ff60 0x7fed74032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fed780096a0 tx=0x7fed78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:35.677 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.675+0000 7fed7e7fc700 1 -- 172.21.15.187:0/3115523871 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed84028440 con 0x7fed880601f0 2021-11-20T05:30:35.880 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.881+0000 7fed8e8e7700 1 -- 172.21.15.187:0/3115523871 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed8810c850 con 0x7fed880601f0 2021-11-20T05:30:35.881 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.882+0000 7fed7e7fc700 1 -- 172.21.15.187:0/3115523871 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fed8402d2d0 con 0x7fed880601f0 2021-11-20T05:30:35.882 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:30:35.882 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"b33f8c4a-49c2-11ec-8c2c-001a4aab830c","modified":"2021-11-20T05:28:52.613926Z","created":"2021-11-20T05:28:52.613926Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi058","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:3300","nonce":0},{"type":"v1","addr":"172.21.15.58:6789","nonce":0}]},"addr":"172.21.15.58:6789/0","public_addr":"172.21.15.58:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T05:30:35.886 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.883+0000 7fed8e8e7700 1 -- 172.21.15.187:0/3115523871 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fed7402ff60 msgr2=0x7fed74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:35.886 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.883+0000 7fed8e8e7700 1 --2- 172.21.15.187:0/3115523871 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fed7402ff60 0x7fed74032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fed780096a0 tx=0x7fed78006b40).stop 2021-11-20T05:30:35.887 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.883+0000 7fed8e8e7700 1 -- 172.21.15.187:0/3115523871 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fed880601f0 msgr2=0x7fed8810b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:35.887 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.883+0000 7fed8e8e7700 1 --2- 172.21.15.187:0/3115523871 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fed880601f0 0x7fed8810b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fed84009f80 tx=0x7fed84009b30).stop 2021-11-20T05:30:35.887 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.883+0000 7fed8e8e7700 1 -- 172.21.15.187:0/3115523871 shutdown_connections 2021-11-20T05:30:35.888 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.883+0000 7fed8e8e7700 1 --2- 172.21.15.187:0/3115523871 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fed7402ff60 0x7fed74032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:35.888 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.883+0000 7fed8e8e7700 1 --2- 172.21.15.187:0/3115523871 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fed880601f0 0x7fed8810b750 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:35.889 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.883+0000 7fed8e8e7700 1 -- 172.21.15.187:0/3115523871 >> 172.21.15.187:0/3115523871 conn(0x7fed880f09a0 msgr2=0x7fed880f1520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:35.889 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.883+0000 7fed8e8e7700 1 -- 172.21.15.187:0/3115523871 shutdown_connections 2021-11-20T05:30:35.890 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:35.883+0000 7fed8e8e7700 1 -- 172.21.15.187:0/3115523871 wait complete. 2021-11-20T05:30:35.890 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2021-11-20T05:30:36.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:36 smithi058 conmon[33537]: audit 2021-11-20T05:30:35.882419+0000 mon.smithi058 ( 2021-11-20T05:30:36.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:36 smithi058 conmon[33537]: mon.0) 134 : audit [DBG] from='client.? 172.21.15.187:0/3115523871' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T05:30:37.405 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T05:30:37.405 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T05:30:37.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:37 smithi058 conmon[33537]: cluster 2021-11-20T05:30:36.173552+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:30:37.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:37 smithi058 conmon[33537]: mgr.14164) 24 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:38.990 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.989+0000 7f8dbb30f700 1 -- 172.21.15.187:0/269469931 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8db40f5410 msgr2=0x7f8db40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:38.990 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.989+0000 7f8dbb30f700 1 --2- 172.21.15.187:0/269469931 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8db40f5410 0x7f8db40f5830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f8da4009960 tx=0x7f8da4004d10).stop 2021-11-20T05:30:38.990 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.989+0000 7f8dbb30f700 1 -- 172.21.15.187:0/269469931 shutdown_connections 2021-11-20T05:30:38.991 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.989+0000 7f8dbb30f700 1 --2- 172.21.15.187:0/269469931 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8db40f5410 0x7f8db40f5830 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:38.991 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.989+0000 7f8dbb30f700 1 -- 172.21.15.187:0/269469931 >> 172.21.15.187:0/269469931 conn(0x7f8db40f0b70 msgr2=0x7f8db40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:38.992 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.989+0000 7f8dbb30f700 1 -- 172.21.15.187:0/269469931 shutdown_connections 2021-11-20T05:30:38.992 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.989+0000 7f8dbb30f700 1 -- 172.21.15.187:0/269469931 wait complete. 2021-11-20T05:30:38.992 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.990+0000 7f8dbb30f700 1 Processor -- start 2021-11-20T05:30:38.992 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.990+0000 7f8dbb30f700 1 -- start start 2021-11-20T05:30:38.993 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.990+0000 7f8dbb30f700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8db40f5410 0x7f8db41094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:38.993 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.990+0000 7f8dbb30f700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db4060220 con 0x7f8db40f5410 2021-11-20T05:30:38.993 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.990+0000 7f8db90ab700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8db40f5410 0x7f8db41094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:38.993 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.990+0000 7f8db90ab700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8db40f5410 0x7f8db41094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.187:46068/0 (socket says 172.21.15.187:46068) 2021-11-20T05:30:38.994 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.990+0000 7f8db90ab700 1 -- 172.21.15.187:0/1090737942 learned_addr learned my addr 172.21.15.187:0/1090737942 (peer_addr_for_me v2:172.21.15.187:0/0) 2021-11-20T05:30:38.994 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.990+0000 7f8db90ab700 1 -- 172.21.15.187:0/1090737942 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da4009640 con 0x7f8db40f5410 2021-11-20T05:30:38.994 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.990+0000 7f8db90ab700 1 --2- 172.21.15.187:0/1090737942 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8db40f5410 0x7f8db41094b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f8da4004a10 tx=0x7f8da4004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:38.994 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.991+0000 7f8db1ffb700 1 -- 172.21.15.187:0/1090737942 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8da4018070 con 0x7f8db40f5410 2021-11-20T05:30:38.995 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.991+0000 7f8db1ffb700 1 -- 172.21.15.187:0/1090737942 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8da4007e70 con 0x7f8db40f5410 2021-11-20T05:30:38.995 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.991+0000 7f8db1ffb700 1 -- 172.21.15.187:0/1090737942 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8da401d420 con 0x7f8db40f5410 2021-11-20T05:30:38.995 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.991+0000 7f8dbb30f700 1 -- 172.21.15.187:0/1090737942 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db4060420 con 0x7f8db40f5410 2021-11-20T05:30:38.995 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.991+0000 7f8dbb30f700 1 -- 172.21.15.187:0/1090737942 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db4109b60 con 0x7f8db40f5410 2021-11-20T05:30:38.996 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.991+0000 7f8db1ffb700 1 -- 172.21.15.187:0/1090737942 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8da4013630 con 0x7f8db40f5410 2021-11-20T05:30:38.997 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.992+0000 7f8dbb30f700 1 -- 172.21.15.187:0/1090737942 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db4059100 con 0x7f8db40f5410 2021-11-20T05:30:38.997 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.992+0000 7f8db1ffb700 1 --2- 172.21.15.187:0/1090737942 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8d9c02ffb0 0x7f8d9c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:38.997 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.992+0000 7f8db1ffb700 1 -- 172.21.15.187:0/1090737942 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8da40415f0 con 0x7f8db40f5410 2021-11-20T05:30:38.997 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.992+0000 7f8db88aa700 1 --2- 172.21.15.187:0/1090737942 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8d9c02ffb0 0x7f8d9c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:38.998 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.992+0000 7f8db88aa700 1 --2- 172.21.15.187:0/1090737942 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8d9c02ffb0 0x7f8d9c032470 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f8da80096a0 tx=0x7f8da8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:38.998 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:38.996+0000 7f8db1ffb700 1 -- 172.21.15.187:0/1090737942 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8da401a2d0 con 0x7f8db40f5410 2021-11-20T05:30:39.199 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:39.200+0000 7f8dbb30f700 1 -- 172.21.15.187:0/1090737942 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8db4045ad0 con 0x7f8db40f5410 2021-11-20T05:30:39.200 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:39.201+0000 7f8db1ffb700 1 -- 172.21.15.187:0/1090737942 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8da400b870 con 0x7f8db40f5410 2021-11-20T05:30:39.200 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:30:39.201 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"b33f8c4a-49c2-11ec-8c2c-001a4aab830c","modified":"2021-11-20T05:28:52.613926Z","created":"2021-11-20T05:28:52.613926Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi058","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:3300","nonce":0},{"type":"v1","addr":"172.21.15.58:6789","nonce":0}]},"addr":"172.21.15.58:6789/0","public_addr":"172.21.15.58:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T05:30:39.202 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:39.202+0000 7f8dbb30f700 1 -- 172.21.15.187:0/1090737942 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8d9c02ffb0 msgr2=0x7f8d9c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:39.202 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:39.202+0000 7f8dbb30f700 1 --2- 172.21.15.187:0/1090737942 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8d9c02ffb0 0x7f8d9c032470 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f8da80096a0 tx=0x7f8da8006b40).stop 2021-11-20T05:30:39.203 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:39.202+0000 7f8dbb30f700 1 -- 172.21.15.187:0/1090737942 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8db40f5410 msgr2=0x7f8db41094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:39.203 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:39.202+0000 7f8dbb30f700 1 --2- 172.21.15.187:0/1090737942 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8db40f5410 0x7f8db41094b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f8da4004a10 tx=0x7f8da4004d10).stop 2021-11-20T05:30:39.203 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:39.202+0000 7f8dbb30f700 1 -- 172.21.15.187:0/1090737942 shutdown_connections 2021-11-20T05:30:39.203 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:39.202+0000 7f8dbb30f700 1 --2- 172.21.15.187:0/1090737942 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8d9c02ffb0 0x7f8d9c032470 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:39.203 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:39.202+0000 7f8dbb30f700 1 --2- 172.21.15.187:0/1090737942 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8db40f5410 0x7f8db41094b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:39.204 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:39.203+0000 7f8dbb30f700 1 -- 172.21.15.187:0/1090737942 >> 172.21.15.187:0/1090737942 conn(0x7f8db40f0b70 msgr2=0x7f8db40f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:39.204 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:39.203+0000 7f8dbb30f700 1 -- 172.21.15.187:0/1090737942 shutdown_connections 2021-11-20T05:30:39.204 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:39.203+0000 7f8dbb30f700 1 -- 172.21.15.187:0/1090737942 wait complete. 2021-11-20T05:30:39.204 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2021-11-20T05:30:39.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:39 smithi058 conmon[33537]: cluster 2021-11-20T05:30:38.173840+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:30:39.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:39 smithi058 conmon[33537]: ) 25 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:39.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:39 smithi058 conmon[33537]: audit 2021-11-20T05:30:39.201444+0000 mon.smithi058 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.187:0/1090737942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T05:30:40.655 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T05:30:40.656 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T05:30:41.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:41 smithi058 conmon[33537]: cluster 2021-11-20T05:30:40.174108+0000 mgr.smithi058.zagdlk (mgr.14164) 26 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:41.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:41 smithi058 conmon[33537]: audit 2021-11-20T05:30:40.892443+0000 mon.smithi058 (mon.0) 136 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:41.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:41 smithi058 conmon[33537]: audit 2021-11-20T05:30:40.892979+0000 mon.smithi058 (mon.0) 137 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:30:42.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:42 smithi058 conmon[33537]: audit 2021-11-20T05:30:41.242583+0000 2021-11-20T05:30:42.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:42 smithi058 conmon[33537]: mon.smithi058 (mon.0) 138 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:42.742 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.739+0000 7f42bd4df700 1 -- 172.21.15.187:0/2817768646 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f42b80f51a0 msgr2=0x7f42b80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:42.742 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.739+0000 7f42bd4df700 1 --2- 172.21.15.187:0/2817768646 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f42b80f51a0 0x7f42b80f55c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f42a8009fb0 tx=0x7f42a8009b30).stop 2021-11-20T05:30:42.742 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.739+0000 7f42bd4df700 1 -- 172.21.15.187:0/2817768646 shutdown_connections 2021-11-20T05:30:42.743 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.739+0000 7f42bd4df700 1 --2- 172.21.15.187:0/2817768646 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f42b80f51a0 0x7f42b80f55c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:42.743 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.739+0000 7f42bd4df700 1 -- 172.21.15.187:0/2817768646 >> 172.21.15.187:0/2817768646 conn(0x7f42b80f09a0 msgr2=0x7f42b80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:42.743 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.739+0000 7f42bd4df700 1 -- 172.21.15.187:0/2817768646 shutdown_connections 2021-11-20T05:30:42.743 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.739+0000 7f42bd4df700 1 -- 172.21.15.187:0/2817768646 wait complete. 2021-11-20T05:30:42.744 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.740+0000 7f42bd4df700 1 Processor -- start 2021-11-20T05:30:42.744 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.740+0000 7f42bd4df700 1 -- start start 2021-11-20T05:30:42.744 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.740+0000 7f42bd4df700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f42b80f51a0 0x7f42b81091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:42.744 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.740+0000 7f42bd4df700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b8060240 con 0x7f42b80f51a0 2021-11-20T05:30:42.744 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.740+0000 7f42b7fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f42b80f51a0 0x7f42b81091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:42.745 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.740+0000 7f42b7fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f42b80f51a0 0x7f42b81091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.187:46074/0 (socket says 172.21.15.187:46074) 2021-11-20T05:30:42.745 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.740+0000 7f42b7fff700 1 -- 172.21.15.187:0/1260680163 learned_addr learned my addr 172.21.15.187:0/1260680163 (peer_addr_for_me v2:172.21.15.187:0/0) 2021-11-20T05:30:42.745 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.740+0000 7f42b7fff700 1 -- 172.21.15.187:0/1260680163 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a8005040 con 0x7f42b80f51a0 2021-11-20T05:30:42.746 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.741+0000 7f42b7fff700 1 --2- 172.21.15.187:0/1260680163 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f42b80f51a0 0x7f42b81091b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f42a8013510 tx=0x7f42a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:42.746 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.741+0000 7f42b57fa700 1 -- 172.21.15.187:0/1260680163 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42a8009320 con 0x7f42b80f51a0 2021-11-20T05:30:42.746 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.741+0000 7f42b57fa700 1 -- 172.21.15.187:0/1260680163 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42a8013f00 con 0x7f42b80f51a0 2021-11-20T05:30:42.746 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.741+0000 7f42bd4df700 1 -- 172.21.15.187:0/1260680163 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b8060440 con 0x7f42b80f51a0 2021-11-20T05:30:42.747 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.741+0000 7f42b57fa700 1 -- 172.21.15.187:0/1260680163 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42a8014dc0 con 0x7f42b80f51a0 2021-11-20T05:30:42.747 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.741+0000 7f42bd4df700 1 -- 172.21.15.187:0/1260680163 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b8109860 con 0x7f42b80f51a0 2021-11-20T05:30:42.747 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.741+0000 7f42b57fa700 1 -- 172.21.15.187:0/1260680163 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42a801d3d0 con 0x7f42b80f51a0 2021-11-20T05:30:42.747 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.742+0000 7f42b57fa700 1 --2- 172.21.15.187:0/1260680163 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f42a402ff60 0x7f42a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:42.748 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.742+0000 7f42b57fa700 1 -- 172.21.15.187:0/1260680163 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f42a8041010 con 0x7f42b80f51a0 2021-11-20T05:30:42.748 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.742+0000 7f42bd4df700 1 -- 172.21.15.187:0/1260680163 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b8045ad0 con 0x7f42b80f51a0 2021-11-20T05:30:42.748 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.742+0000 7f42affff700 1 --2- 172.21.15.187:0/1260680163 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f42a402ff60 0x7f42a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:42.748 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.742+0000 7f42affff700 1 --2- 172.21.15.187:0/1260680163 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f42a402ff60 0x7f42a4032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f42a00096a0 tx=0x7f42a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:42.749 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.746+0000 7f42b57fa700 1 -- 172.21.15.187:0/1260680163 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42a801db80 con 0x7f42b80f51a0 2021-11-20T05:30:42.961 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.962+0000 7f42bd4df700 1 -- 172.21.15.187:0/1260680163 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42b80fabe0 con 0x7f42b80f51a0 2021-11-20T05:30:42.962 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.962+0000 7f42b57fa700 1 -- 172.21.15.187:0/1260680163 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f42a801c970 con 0x7f42b80f51a0 2021-11-20T05:30:42.962 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:30:42.963 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"b33f8c4a-49c2-11ec-8c2c-001a4aab830c","modified":"2021-11-20T05:28:52.613926Z","created":"2021-11-20T05:28:52.613926Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi058","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:3300","nonce":0},{"type":"v1","addr":"172.21.15.58:6789","nonce":0}]},"addr":"172.21.15.58:6789/0","public_addr":"172.21.15.58:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T05:30:42.964 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.964+0000 7f42bd4df700 1 -- 172.21.15.187:0/1260680163 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f42a402ff60 msgr2=0x7f42a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:42.965 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.964+0000 7f42bd4df700 1 --2- 172.21.15.187:0/1260680163 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f42a402ff60 0x7f42a4032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f42a00096a0 tx=0x7f42a0006b40).stop 2021-11-20T05:30:42.965 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.964+0000 7f42bd4df700 1 -- 172.21.15.187:0/1260680163 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f42b80f51a0 msgr2=0x7f42b81091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:42.965 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.964+0000 7f42bd4df700 1 --2- 172.21.15.187:0/1260680163 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f42b80f51a0 0x7f42b81091b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f42a8013510 tx=0x7f42a800b040).stop 2021-11-20T05:30:42.965 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.964+0000 7f42bd4df700 1 -- 172.21.15.187:0/1260680163 shutdown_connections 2021-11-20T05:30:42.965 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.964+0000 7f42bd4df700 1 --2- 172.21.15.187:0/1260680163 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f42a402ff60 0x7f42a4032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:42.966 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.964+0000 7f42bd4df700 1 --2- 172.21.15.187:0/1260680163 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f42b80f51a0 0x7f42b81091b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:42.966 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.964+0000 7f42bd4df700 1 -- 172.21.15.187:0/1260680163 >> 172.21.15.187:0/1260680163 conn(0x7f42b80f09a0 msgr2=0x7f42b80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:42.966 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.964+0000 7f42bd4df700 1 -- 172.21.15.187:0/1260680163 shutdown_connections 2021-11-20T05:30:42.966 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:42.964+0000 7f42bd4df700 1 -- 172.21.15.187:0/1260680163 wait complete. 2021-11-20T05:30:42.967 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2021-11-20T05:30:43.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:43 smithi058 conmon[33537]: cluster 2021-11-20T05:30:42.174380 2021-11-20T05:30:43.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:43 smithi058 conmon[33537]: +0000 mgr.smithi058.zagdlk (mgr.14164) 27 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:43.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:43 smithi058 conmon[33537]: audit 2021-11-20T05:30:42.339809+0000 mon.smithi058 (mon.0) 139 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:43.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:43 smithi058 conmon[33537]: audit 2021-11-20T05:30:42.963001+0000 mon.smithi058 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.187:0/1260680163' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T05:30:44.934 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T05:30:44.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T05:30:45.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:45 smithi058 conmon[33537]: cluster 2021-11-20T05:30:44.174616+0000 mgr.smithi058.zagdlk 2021-11-20T05:30:45.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:45 smithi058 conmon[33537]: (mgr.14164) 28 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:46.698 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.697+0000 7fc1ab0c1700 1 -- 172.21.15.187:0/683944900 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1a40f5410 msgr2=0x7fc1a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:46.698 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.697+0000 7fc1ab0c1700 1 --2- 172.21.15.187:0/683944900 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1a40f5410 0x7fc1a40f5830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fc198009fb0 tx=0x7fc198009b30).stop 2021-11-20T05:30:46.699 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.697+0000 7fc1ab0c1700 1 -- 172.21.15.187:0/683944900 shutdown_connections 2021-11-20T05:30:46.699 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.697+0000 7fc1ab0c1700 1 --2- 172.21.15.187:0/683944900 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1a40f5410 0x7fc1a40f5830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:46.699 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.697+0000 7fc1ab0c1700 1 -- 172.21.15.187:0/683944900 >> 172.21.15.187:0/683944900 conn(0x7fc1a40f0b70 msgr2=0x7fc1a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:46.699 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.697+0000 7fc1ab0c1700 1 -- 172.21.15.187:0/683944900 shutdown_connections 2021-11-20T05:30:46.700 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.697+0000 7fc1ab0c1700 1 -- 172.21.15.187:0/683944900 wait complete. 2021-11-20T05:30:46.700 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.697+0000 7fc1ab0c1700 1 Processor -- start 2021-11-20T05:30:46.700 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.698+0000 7fc1ab0c1700 1 -- start start 2021-11-20T05:30:46.700 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.698+0000 7fc1ab0c1700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1a40f5410 0x7fc1a4109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:46.700 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.698+0000 7fc1ab0c1700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a4060240 con 0x7fc1a40f5410 2021-11-20T05:30:46.701 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.698+0000 7fc1a8e5d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1a40f5410 0x7fc1a4109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:46.701 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.698+0000 7fc1a8e5d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1a40f5410 0x7fc1a4109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.187:46080/0 (socket says 172.21.15.187:46080) 2021-11-20T05:30:46.701 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.698+0000 7fc1a8e5d700 1 -- 172.21.15.187:0/4235045119 learned_addr learned my addr 172.21.15.187:0/4235045119 (peer_addr_for_me v2:172.21.15.187:0/0) 2021-11-20T05:30:46.701 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.698+0000 7fc1a8e5d700 1 -- 172.21.15.187:0/4235045119 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc198005040 con 0x7fc1a40f5410 2021-11-20T05:30:46.702 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.699+0000 7fc1a8e5d700 1 --2- 172.21.15.187:0/4235045119 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1a40f5410 0x7fc1a4109400 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fc198013510 tx=0x7fc19800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:46.702 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.699+0000 7fc1a17fa700 1 -- 172.21.15.187:0/4235045119 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc198009320 con 0x7fc1a40f5410 2021-11-20T05:30:46.702 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.699+0000 7fc1ab0c1700 1 -- 172.21.15.187:0/4235045119 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a4060440 con 0x7fc1a40f5410 2021-11-20T05:30:46.702 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.699+0000 7fc1a17fa700 1 -- 172.21.15.187:0/4235045119 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc198013f00 con 0x7fc1a40f5410 2021-11-20T05:30:46.702 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.699+0000 7fc1a17fa700 1 -- 172.21.15.187:0/4235045119 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc198014d90 con 0x7fc1a40f5410 2021-11-20T05:30:46.703 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.699+0000 7fc1ab0c1700 1 -- 172.21.15.187:0/4235045119 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a4109ab0 con 0x7fc1a40f5410 2021-11-20T05:30:46.704 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.700+0000 7fc1a17fa700 1 -- 172.21.15.187:0/4235045119 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1980095f0 con 0x7fc1a40f5410 2021-11-20T05:30:46.704 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.700+0000 7fc1ab0c1700 1 -- 172.21.15.187:0/4235045119 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a4045ad0 con 0x7fc1a40f5410 2021-11-20T05:30:46.704 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.700+0000 7fc1a17fa700 1 --2- 172.21.15.187:0/4235045119 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc19402ff60 0x7fc194032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:46.705 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.700+0000 7fc1a17fa700 1 -- 172.21.15.187:0/4235045119 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc198040fd0 con 0x7fc1a40f5410 2021-11-20T05:30:46.705 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.700+0000 7fc1a3fff700 1 --2- 172.21.15.187:0/4235045119 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc19402ff60 0x7fc194032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:46.705 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.700+0000 7fc1a3fff700 1 --2- 172.21.15.187:0/4235045119 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc19402ff60 0x7fc194032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fc1900096a0 tx=0x7fc190006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:46.705 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.705+0000 7fc1a17fa700 1 -- 172.21.15.187:0/4235045119 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc198014420 con 0x7fc1a40f5410 2021-11-20T05:30:46.926 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.927+0000 7fc1ab0c1700 1 -- 172.21.15.187:0/4235045119 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1a40fae60 con 0x7fc1a40f5410 2021-11-20T05:30:46.928 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.928+0000 7fc1a17fa700 1 -- 172.21.15.187:0/4235045119 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc19801d7f0 con 0x7fc1a40f5410 2021-11-20T05:30:46.928 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:30:46.929 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"b33f8c4a-49c2-11ec-8c2c-001a4aab830c","modified":"2021-11-20T05:28:52.613926Z","created":"2021-11-20T05:28:52.613926Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi058","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:3300","nonce":0},{"type":"v1","addr":"172.21.15.58:6789","nonce":0}]},"addr":"172.21.15.58:6789/0","public_addr":"172.21.15.58:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T05:30:46.930 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.930+0000 7fc1ab0c1700 1 -- 172.21.15.187:0/4235045119 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc19402ff60 msgr2=0x7fc194032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:46.930 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.930+0000 7fc1ab0c1700 1 --2- 172.21.15.187:0/4235045119 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc19402ff60 0x7fc194032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fc1900096a0 tx=0x7fc190006b40).stop 2021-11-20T05:30:46.930 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.930+0000 7fc1ab0c1700 1 -- 172.21.15.187:0/4235045119 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1a40f5410 msgr2=0x7fc1a4109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:46.931 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.930+0000 7fc1ab0c1700 1 --2- 172.21.15.187:0/4235045119 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1a40f5410 0x7fc1a4109400 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fc198013510 tx=0x7fc19800b040).stop 2021-11-20T05:30:46.931 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.930+0000 7fc1ab0c1700 1 -- 172.21.15.187:0/4235045119 shutdown_connections 2021-11-20T05:30:46.931 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.930+0000 7fc1ab0c1700 1 --2- 172.21.15.187:0/4235045119 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc19402ff60 0x7fc194032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:46.931 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.930+0000 7fc1ab0c1700 1 --2- 172.21.15.187:0/4235045119 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1a40f5410 0x7fc1a4109400 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:46.931 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.930+0000 7fc1ab0c1700 1 -- 172.21.15.187:0/4235045119 >> 172.21.15.187:0/4235045119 conn(0x7fc1a40f0b70 msgr2=0x7fc1a40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:46.932 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.930+0000 7fc1ab0c1700 1 -- 172.21.15.187:0/4235045119 shutdown_connections 2021-11-20T05:30:46.932 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:46.930+0000 7fc1ab0c1700 1 -- 172.21.15.187:0/4235045119 wait complete. 2021-11-20T05:30:46.932 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2021-11-20T05:30:46.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: audit 2021-11-20T05:30:45.601842+0000 mon.smithi058 (mon.0) 141 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:46.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: audit 2021-11-20T05:30:45.782245+0000 mon.smithi058 (mon.0) 142 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:46.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: cluster 2021-11-20T05:30:46.175002+0000 mgr.smithi058.zagdlk (mgr.14164) 29 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:46.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: audit 2021-11-20T05:30:46.327106+0000 2021-11-20T05:30:46.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: mon.smithi058 (mon.0) 143 : audit [INF] 2021-11-20T05:30:46.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:46.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:30:46.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: -20T05:30:46.329687+0000 2021-11-20T05:30:46.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: mon.smithi058 (mon.0) 144 : audit [INF] 2021-11-20T05:30:46.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:46.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: audit 2021-11-20T05:30: 2021-11-20T05:30:46.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: 46.330873+0000 mon.smithi058 (mon.0) 145 2021-11-20T05:30:46.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi187", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T05:30:46.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: audit 2021-11-20 2021-11-20T05:30:46.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: T05:30:46.332697+0000 mon.smithi058 ( 2021-11-20T05:30:46.975 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: mon.0) 146 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi187", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2021-11-20T05:30:46.975 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:30:46.975 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: 11-20T05:30:46.333496+0000 2021-11-20T05:30:46.976 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: mon.smithi058 (mon.0) 147 : audit [DBG] 2021-11-20T05:30:46.976 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:30:46.977 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: cephadm 2021-11-20T05:30:46 2021-11-20T05:30:46.977 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: .334254+0000 mgr.smithi058.zagdlk (mgr.14164) 30 : 2021-11-20T05:30:46.978 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:46 smithi058 conmon[33537]: cephadm [INF] Deploying daemon crash.smithi187 on smithi187 2021-11-20T05:30:48.219 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:47 smithi058 conmon[33537]: audit 2021-11-20T05:30:46.928369+0000 mon.smithi058 (mon.0 2021-11-20T05:30:48.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:47 smithi058 conmon[33537]: ) 148 : audit [DBG] from='client.? 172.21.15.187:0/4235045119' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T05:30:49.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:48 smithi058 conmon[33537]: cluster 2021-11-20T05:30:48.175362+0000 mgr.smithi058.zagdlk (mgr 2021-11-20T05:30:49.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:48 smithi058 conmon[33537]: .14164) 31 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:50.284 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T05:30:50.285 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T05:30:51.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:51 smithi058 conmon[33537]: cluster 2021-11-20T05:30:50.175594+0000 mgr.smithi058.zagdlk (mgr.14164) 32 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:53.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:53 smithi058 conmon[33537]: cluster 2021-11-20T05:30:52.175916+0000 mgr.smithi058.zagdlk (mgr.14164) 33 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:53.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:53 smithi058 conmon[33537]: audit 2021-11-20T05:30:53.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:53 smithi058 conmon[33537]: 2021-11-20T05:30:52.927921+0000 mon.smithi058 (mon.0) 149 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:30:53.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:53 smithi058 conmon[33537]: audit 2021-11-20T05:30:52.929331+0000 mon.smithi058 (mon.0) 150 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi187.omkctn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T05:30:53.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:53 smithi058 conmon[33537]: audit 2021-11-20T05:30:52.931339+0000 mon.smithi058 (mon 2021-11-20T05:30:53.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:53 smithi058 conmon[33537]: .0) 151 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi187.omkctn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-11-20T05:30:53.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:53 smithi058 conmon[33537]: audit 2021-11- 2021-11-20T05:30:53.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:53 smithi058 conmon[33537]: 20T05:30:52.932244+0000 mon.smithi058 ( 2021-11-20T05:30:53.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:53 smithi058 conmon[33537]: mon.0) 152 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T05:30:53.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:53 smithi058 conmon[33537]: audit 2021-11-20T05 2021-11-20T05:30:53.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:53 smithi058 conmon[33537]: :30:52.933361+0000 mon.smithi058 (mon.0) 153 : audit 2021-11-20T05:30:53.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:53 smithi058 conmon[33537]: [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:30:54.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:54 smithi058 conmon[33537]: cephadm 2021-11-20T05:30:52.934157+0000 mgr.smithi058.zagdlk (mgr.14164) 34 : cephadm [INF] Deploying daemon mgr.smithi187.omkctn on smithi187 2021-11-20T05:30:55.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:55 smithi058 conmon[33537]: cluster 2021-11-20T05:30:54.176174+0000 mgr.smithi058.zagdlk (mgr.14164) 35 : cluster 2021-11-20T05:30:55.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:55 smithi058 conmon[33537]: [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:57.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:57 smithi058 conmon[33537]: cluster 2021-11-20T05:30:56.176540+0000 mgr.smithi058.zagdlk (mgr.14164) 36 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:58.718 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.671+0000 7f0ad225d700 1 -- 172.21.15.187:0/2116197508 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0acc0f78e0 msgr2=0x7f0acc0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:58.718 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.671+0000 7f0ad225d700 1 --2- 172.21.15.187:0/2116197508 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0acc0f78e0 0x7f0acc0f7d00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f0ac8009fb0 tx=0x7f0ac8009b30).stop 2021-11-20T05:30:58.718 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.671+0000 7f0ad225d700 1 -- 172.21.15.187:0/2116197508 shutdown_connections 2021-11-20T05:30:58.719 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.671+0000 7f0ad225d700 1 --2- 172.21.15.187:0/2116197508 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0acc0f78e0 0x7f0acc0f7d00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:58.719 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.671+0000 7f0ad225d700 1 -- 172.21.15.187:0/2116197508 >> 172.21.15.187:0/2116197508 conn(0x7f0acc0f0120 msgr2=0x7f0acc0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:58.719 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.671+0000 7f0ad225d700 1 -- 172.21.15.187:0/2116197508 shutdown_connections 2021-11-20T05:30:58.719 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.671+0000 7f0ad225d700 1 -- 172.21.15.187:0/2116197508 wait complete. 2021-11-20T05:30:58.720 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.672+0000 7f0ad225d700 1 Processor -- start 2021-11-20T05:30:58.720 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.672+0000 7f0ad225d700 1 -- start start 2021-11-20T05:30:58.720 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.672+0000 7f0ad225d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0acc0f78e0 0x7f0acc1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:58.720 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.672+0000 7f0ad225d700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc1069d0 con 0x7f0acc0f78e0 2021-11-20T05:30:58.721 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.672+0000 7f0ad125b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0acc0f78e0 0x7f0acc1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:58.721 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.672+0000 7f0ad125b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0acc0f78e0 0x7f0acc1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.187:46086/0 (socket says 172.21.15.187:46086) 2021-11-20T05:30:58.721 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.672+0000 7f0ad125b700 1 -- 172.21.15.187:0/606002482 learned_addr learned my addr 172.21.15.187:0/606002482 (peer_addr_for_me v2:172.21.15.187:0/0) 2021-11-20T05:30:58.721 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.672+0000 7f0ad125b700 1 -- 172.21.15.187:0/606002482 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac8005040 con 0x7f0acc0f78e0 2021-11-20T05:30:58.721 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.672+0000 7f0ad125b700 1 --2- 172.21.15.187:0/606002482 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0acc0f78e0 0x7f0acc1064e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f0ac8013510 tx=0x7f0ac800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:58.722 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.673+0000 7f0ac1ffb700 1 -- 172.21.15.187:0/606002482 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ac8009320 con 0x7f0acc0f78e0 2021-11-20T05:30:58.722 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.673+0000 7f0ad225d700 1 -- 172.21.15.187:0/606002482 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0acc106bd0 con 0x7f0acc0f78e0 2021-11-20T05:30:58.722 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.673+0000 7f0ac1ffb700 1 -- 172.21.15.187:0/606002482 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ac8013f00 con 0x7f0acc0f78e0 2021-11-20T05:30:58.722 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.673+0000 7f0ac1ffb700 1 -- 172.21.15.187:0/606002482 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ac8014dc0 con 0x7f0acc0f78e0 2021-11-20T05:30:58.723 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.673+0000 7f0ad225d700 1 -- 172.21.15.187:0/606002482 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0acc107060 con 0x7f0acc0f78e0 2021-11-20T05:30:58.723 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.674+0000 7f0ac1ffb700 1 -- 172.21.15.187:0/606002482 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ac80095f0 con 0x7f0acc0f78e0 2021-11-20T05:30:58.723 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.674+0000 7f0ad225d700 1 -- 172.21.15.187:0/606002482 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0acc045ad0 con 0x7f0acc0f78e0 2021-11-20T05:30:58.723 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.674+0000 7f0ac1ffb700 1 --2- 172.21.15.187:0/606002482 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0ab802ff60 0x7f0ab8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:30:58.723 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.674+0000 7f0ac1ffb700 1 -- 172.21.15.187:0/606002482 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ac8041020 con 0x7f0acc0f78e0 2021-11-20T05:30:58.724 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.674+0000 7f0ad0a5a700 1 --2- 172.21.15.187:0/606002482 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0ab802ff60 0x7f0ab8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:30:58.724 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.675+0000 7f0ad0a5a700 1 --2- 172.21.15.187:0/606002482 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0ab802ff60 0x7f0ab8032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f0abc0096a0 tx=0x7f0abc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:30:58.724 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.679+0000 7f0ac1ffb700 1 -- 172.21.15.187:0/606002482 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ac8014420 con 0x7f0acc0f78e0 2021-11-20T05:30:58.887 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.888+0000 7f0ad225d700 1 -- 172.21.15.187:0/606002482 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0acc107eb0 con 0x7f0acc0f78e0 2021-11-20T05:30:58.888 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.889+0000 7f0ac1ffb700 1 -- 172.21.15.187:0/606002482 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ac801c920 con 0x7f0acc0f78e0 2021-11-20T05:30:58.888 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:30:58.889 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"b33f8c4a-49c2-11ec-8c2c-001a4aab830c","modified":"2021-11-20T05:28:52.613926Z","created":"2021-11-20T05:28:52.613926Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi058","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:3300","nonce":0},{"type":"v1","addr":"172.21.15.58:6789","nonce":0}]},"addr":"172.21.15.58:6789/0","public_addr":"172.21.15.58:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T05:30:58.889 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.890+0000 7f0ad225d700 1 -- 172.21.15.187:0/606002482 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0ab802ff60 msgr2=0x7f0ab8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:58.890 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.890+0000 7f0ad225d700 1 --2- 172.21.15.187:0/606002482 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0ab802ff60 0x7f0ab8032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f0abc0096a0 tx=0x7f0abc006b40).stop 2021-11-20T05:30:58.890 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.890+0000 7f0ad225d700 1 -- 172.21.15.187:0/606002482 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0acc0f78e0 msgr2=0x7f0acc1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:30:58.891 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.890+0000 7f0ad225d700 1 --2- 172.21.15.187:0/606002482 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0acc0f78e0 0x7f0acc1064e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f0ac8013510 tx=0x7f0ac800b040).stop 2021-11-20T05:30:58.891 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.890+0000 7f0ad225d700 1 -- 172.21.15.187:0/606002482 shutdown_connections 2021-11-20T05:30:58.891 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.890+0000 7f0ad225d700 1 --2- 172.21.15.187:0/606002482 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0ab802ff60 0x7f0ab8032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:58.891 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.890+0000 7f0ad225d700 1 --2- 172.21.15.187:0/606002482 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0acc0f78e0 0x7f0acc1064e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:30:58.891 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.890+0000 7f0ad225d700 1 -- 172.21.15.187:0/606002482 >> 172.21.15.187:0/606002482 conn(0x7f0acc0f0120 msgr2=0x7f0acc0f0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:30:58.892 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.890+0000 7f0ad225d700 1 -- 172.21.15.187:0/606002482 shutdown_connections 2021-11-20T05:30:58.893 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:30:58.890+0000 7f0ad225d700 1 -- 172.21.15.187:0/606002482 wait complete. 2021-11-20T05:30:58.894 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2021-11-20T05:30:59.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:59 smithi058 conmon[33537]: cluster 2021-11-20T05:30:58.176998+0000 mgr.smithi058.zagdlk (mgr.14164) 37 : cluster 2021-11-20T05:30:59.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:59 smithi058 conmon[33537]: [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:30:59.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:30:59 smithi058 conmon[33537]: audit 2021-11-20T05:30:58.889254+0000 mon.smithi058 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.187:0/606002482' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T05:31:00.262 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T05:31:00.263 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T05:31:01.219 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:00 smithi058 conmon[33537]: audit 2021-11-20T05:30:59.794090+0000 mon.smithi058 (mon.0) 155 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:01.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:00 smithi058 conmon[33537]: audit 2021-11-20T05:31:01.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:00 smithi058 conmon[33537]: 2021-11-20T05:30:59.795423+0000 mon.smithi058 (mon.0) 156 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T05:31:01.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:00 smithi058 conmon[33537]: audit 2021-11-20T05:30:59.796373+0000 mon.smithi058 (mon.0) 157 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:01.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:00 smithi058 conmon[33537]: cephadm 2021 2021-11-20T05:31:01.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:00 smithi058 conmon[33537]: -11-20T05:30:59.797195+0000 mgr.smithi058.zagdlk (mgr.14164) 38 : cephadm [INF] 2021-11-20T05:31:01.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:00 smithi058 conmon[33537]: Deploying daemon mon.smithi187 on smithi187 2021-11-20T05:31:01.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:00 smithi058 conmon[33537]: cluster 2021-11-20T05:31: 2021-11-20T05:31:01.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:00 smithi058 conmon[33537]: 00.177291+0000 mgr.smithi058.zagdlk (mgr.14164) 39 : 2021-11-20T05:31:01.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:00 smithi058 conmon[33537]: cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:01.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:00 smithi058 conmon[33537]: audit 2021-11-20T 2021-11-20T05:31:01.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:00 smithi058 conmon[33537]: 05:31:00.355074+0000 mon.smithi058 (mon 2021-11-20T05:31:01.223 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:00 smithi058 conmon[33537]: .0) 158 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:31:01.223 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:00 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:31:01.224 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:00 smithi058 conmon[33537]: -20T05:31:00.362714+0000 mon.smithi058 2021-11-20T05:31:01.224 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:00 smithi058 conmon[33537]: (mon.0) 159 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:31:01.224 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:00 smithi058 conmon[33537]: 2021-11-20T05:31:02.566 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.565+0000 7fb55cd1a700 1 -- 172.21.15.187:0/159120323 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5580f7b60 msgr2=0x7fb5580f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:02.567 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.565+0000 7fb55cd1a700 1 --2- 172.21.15.187:0/159120323 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5580f7b60 0x7fb5580f7f80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fb548009fb0 tx=0x7fb548009b30).stop 2021-11-20T05:31:02.567 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.565+0000 7fb55cd1a700 1 -- 172.21.15.187:0/159120323 shutdown_connections 2021-11-20T05:31:02.567 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.565+0000 7fb55cd1a700 1 --2- 172.21.15.187:0/159120323 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5580f7b60 0x7fb5580f7f80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:02.567 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.565+0000 7fb55cd1a700 1 -- 172.21.15.187:0/159120323 >> 172.21.15.187:0/159120323 conn(0x7fb5580f02e0 msgr2=0x7fb5580f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:31:02.568 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.565+0000 7fb55cd1a700 1 -- 172.21.15.187:0/159120323 shutdown_connections 2021-11-20T05:31:02.568 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.565+0000 7fb55cd1a700 1 -- 172.21.15.187:0/159120323 wait complete. 2021-11-20T05:31:02.568 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.566+0000 7fb55cd1a700 1 Processor -- start 2021-11-20T05:31:02.568 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.566+0000 7fb55cd1a700 1 -- start start 2021-11-20T05:31:02.569 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.566+0000 7fb55cd1a700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5580f7b60 0x7fb5580f7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:31:02.569 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.566+0000 7fb55cd1a700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5580f58b0 con 0x7fb5580f7b60 2021-11-20T05:31:02.569 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.566+0000 7fb55659c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5580f7b60 0x7fb5580f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:31:02.569 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.566+0000 7fb55659c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5580f7b60 0x7fb5580f7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.187:46096/0 (socket says 172.21.15.187:46096) 2021-11-20T05:31:02.570 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.566+0000 7fb55659c700 1 -- 172.21.15.187:0/2373664289 learned_addr learned my addr 172.21.15.187:0/2373664289 (peer_addr_for_me v2:172.21.15.187:0/0) 2021-11-20T05:31:02.570 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.567+0000 7fb55659c700 1 -- 172.21.15.187:0/2373664289 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb548005040 con 0x7fb5580f7b60 2021-11-20T05:31:02.570 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.567+0000 7fb55659c700 1 --2- 172.21.15.187:0/2373664289 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5580f7b60 0x7fb5580f7430 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fb548013510 tx=0x7fb54800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:31:02.570 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.567+0000 7fb546ffd700 1 -- 172.21.15.187:0/2373664289 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb548009320 con 0x7fb5580f7b60 2021-11-20T05:31:02.571 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.567+0000 7fb55cd1a700 1 -- 172.21.15.187:0/2373664289 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5580f5a50 con 0x7fb5580f7b60 2021-11-20T05:31:02.571 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.567+0000 7fb546ffd700 1 -- 172.21.15.187:0/2373664289 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb548013f00 con 0x7fb5580f7b60 2021-11-20T05:31:02.571 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.567+0000 7fb546ffd700 1 -- 172.21.15.187:0/2373664289 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb548014d90 con 0x7fb5580f7b60 2021-11-20T05:31:02.571 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.567+0000 7fb55cd1a700 1 -- 172.21.15.187:0/2373664289 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5580f5ee0 con 0x7fb5580f7b60 2021-11-20T05:31:02.572 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.568+0000 7fb546ffd700 1 -- 172.21.15.187:0/2373664289 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5480263a0 con 0x7fb5580f7b60 2021-11-20T05:31:02.572 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.568+0000 7fb546ffd700 1 --2- 172.21.15.187:0/2373664289 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb54002ffb0 0x7fb540032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:31:02.573 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.568+0000 7fb55cd1a700 1 -- 172.21.15.187:0/2373664289 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb558059100 con 0x7fb5580f7b60 2021-11-20T05:31:02.573 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.568+0000 7fb546ffd700 1 -- 172.21.15.187:0/2373664289 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb548041060 con 0x7fb5580f7b60 2021-11-20T05:31:02.573 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.569+0000 7fb555d9b700 1 --2- 172.21.15.187:0/2373664289 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb54002ffb0 0x7fb540032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:31:02.573 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.569+0000 7fb555d9b700 1 --2- 172.21.15.187:0/2373664289 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb54002ffb0 0x7fb540032470 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fb54c0096a0 tx=0x7fb54c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:31:02.574 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.573+0000 7fb546ffd700 1 -- 172.21.15.187:0/2373664289 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb548012920 con 0x7fb5580f7b60 2021-11-20T05:31:02.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:02 smithi187 systemd[1]: Starting Ceph mon.smithi187 for b33f8c4a-49c2-11ec-8c2c-001a4aab830c... 2021-11-20T05:31:02.797 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.797+0000 7fb55cd1a700 1 -- 172.21.15.187:0/2373664289 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5580fbf60 con 0x7fb5580f7b60 2021-11-20T05:31:02.798 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.798+0000 7fb546ffd700 1 -- 172.21.15.187:0/2373664289 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb54802b020 con 0x7fb5580f7b60 2021-11-20T05:31:02.798 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:31:02.798 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"b33f8c4a-49c2-11ec-8c2c-001a4aab830c","modified":"2021-11-20T05:28:52.613926Z","created":"2021-11-20T05:28:52.613926Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi058","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:3300","nonce":0},{"type":"v1","addr":"172.21.15.58:6789","nonce":0}]},"addr":"172.21.15.58:6789/0","public_addr":"172.21.15.58:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T05:31:02.800 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.800+0000 7fb55cd1a700 1 -- 172.21.15.187:0/2373664289 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb54002ffb0 msgr2=0x7fb540032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:02.800 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.800+0000 7fb55cd1a700 1 --2- 172.21.15.187:0/2373664289 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb54002ffb0 0x7fb540032470 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fb54c0096a0 tx=0x7fb54c006b40).stop 2021-11-20T05:31:02.800 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.800+0000 7fb55cd1a700 1 -- 172.21.15.187:0/2373664289 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5580f7b60 msgr2=0x7fb5580f7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:02.800 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.800+0000 7fb55cd1a700 1 --2- 172.21.15.187:0/2373664289 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5580f7b60 0x7fb5580f7430 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fb548013510 tx=0x7fb54800b040).stop 2021-11-20T05:31:02.801 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.800+0000 7fb55cd1a700 1 -- 172.21.15.187:0/2373664289 shutdown_connections 2021-11-20T05:31:02.801 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.800+0000 7fb55cd1a700 1 --2- 172.21.15.187:0/2373664289 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb54002ffb0 0x7fb540032470 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:02.801 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.800+0000 7fb55cd1a700 1 --2- 172.21.15.187:0/2373664289 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb5580f7b60 0x7fb5580f7430 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:02.801 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.800+0000 7fb55cd1a700 1 -- 172.21.15.187:0/2373664289 >> 172.21.15.187:0/2373664289 conn(0x7fb5580f02e0 msgr2=0x7fb5580f0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:31:02.802 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.800+0000 7fb55cd1a700 1 -- 172.21.15.187:0/2373664289 shutdown_connections 2021-11-20T05:31:02.802 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:02.800+0000 7fb55cd1a700 1 -- 172.21.15.187:0/2373664289 wait complete. 2021-11-20T05:31:02.802 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2021-11-20T05:31:02.981 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:02 smithi187 podman[36319]: 2021-11-20T05:31:03.357 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.050+0000 7f4f64aba700 0 set uid:gid to 167:167 (ceph:ceph) 2021-11-20T05:31:03.357 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.050+0000 7f4f64aba700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2021-11-20T05:31:03.357 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.051+0000 7f4f64aba700 0 pidfile_write: ignore empty --pid-file 2021-11-20T05:31:03.358 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 0 load: jerasure load: lrc load: isa 2021-11-20T05:31:03.358 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 bash[36214]: 6f61445461b0f0135f80544325440e4492899ca38a968b80e70d35401d3bc171 2021-11-20T05:31:03.358 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: RocksDB version: 6.8.1 2021-11-20T05:31:03.359 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.359 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-11-20T05:31:03.359 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Compile date May 13 2021 2021-11-20T05:31:03.360 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: DB SUMMARY 2021-11-20T05:31:03.360 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.360 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: CURRENT file: CURRENT 2021-11-20T05:31:03.360 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.361 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: IDENTITY file: IDENTITY 2021-11-20T05:31:03.361 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.361 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.361 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2021-11-20T05:31:03.362 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.362 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi187/store.db dir, Total Num: 0, files: 2021-11-20T05:31:03.362 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.363 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi187/store.db: 000003.log size: 511 ; 2021-11-20T05:31:03.363 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.363 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.error_if_exists: 0 2021-11-20T05:31:03.363 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.create_if_missing: 0 2021-11-20T05:31:03.364 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.paranoid_checks: 1 2021-11-20T05:31:03.364 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.env: 0x557e37b791c0 2021-11-20T05:31:03.364 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.fs: Posix File System 2021-11-20T05:31:03.364 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.info_log: 0x557e39745960 2021-11-20T05:31:03.365 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.max_file_opening_threads: 16 2021-11-20T05:31:03.365 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.statistics: (nil) 2021-11-20T05:31:03.365 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.use_fsync: 0 2021-11-20T05:31:03.365 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.max_log_file_size: 0 2021-11-20T05:31:03.366 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-11-20T05:31:03.366 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-11-20T05:31:03.366 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.keep_log_file_num: 1000 2021-11-20T05:31:03.366 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.recycle_log_file_num: 0 2021-11-20T05:31:03.367 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.allow_fallocate: 1 2021-11-20T05:31:03.367 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.allow_mmap_reads: 0 2021-11-20T05:31:03.367 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.allow_mmap_writes: 0 2021-11-20T05:31:03.367 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.use_direct_reads: 0 2021-11-20T05:31:03.367 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-11-20T05:31:03.368 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.create_missing_column_families: 0 2021-11-20T05:31:03.368 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.368 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.db_log_dir: 2021-11-20T05:31:03.368 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi187/store.db 2021-11-20T05:31:03.369 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-11-20T05:31:03.369 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.max_subcompactions: 1 2021-11-20T05:31:03.369 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.max_background_flushes: -1 2021-11-20T05:31:03.369 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-11-20T05:31:03.369 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-11-20T05:31:03.370 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-11-20T05:31:03.370 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.055+0000 7f4f64aba700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-11-20T05:31:03.370 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-11-20T05:31:03.370 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.advise_random_on_open: 1 2021-11-20T05:31:03.371 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.db_write_buffer_size: 0 2021-11-20T05:31:03.371 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.write_buffer_manager: 0x557e3a3ba030 2021-11-20T05:31:03.371 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-11-20T05:31:03.371 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-11-20T05:31:03.372 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-11-20T05:31:03.372 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-11-20T05:31:03.375 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.rate_limiter: (nil) 2021-11-20T05:31:03.376 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-11-20T05:31:03.376 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.wal_recovery_mode: 2 2021-11-20T05:31:03.376 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.enable_thread_tracking: 0 2021-11-20T05:31:03.376 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.enable_pipelined_write: 0 2021-11-20T05:31:03.377 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.unordered_write: 0 2021-11-20T05:31:03.377 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-11-20T05:31:03.377 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-11-20T05:31:03.378 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-11-20T05:31:03.378 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-11-20T05:31:03.378 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.row_cache: None 2021-11-20T05:31:03.378 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.wal_filter: None 2021-11-20T05:31:03.379 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-11-20T05:31:03.379 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.allow_ingest_behind: 0 2021-11-20T05:31:03.379 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.preserve_deletes: 0 2021-11-20T05:31:03.379 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.two_write_queues: 0 2021-11-20T05:31:03.380 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.manual_wal_flush: 0 2021-11-20T05:31:03.380 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.atomic_flush: 0 2021-11-20T05:31:03.380 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-11-20T05:31:03.380 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-11-20T05:31:03.380 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-11-20T05:31:03.381 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.log_readahead_size: 0 2021-11-20T05:31:03.381 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-11-20T05:31:03.381 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_background_jobs: 2 2021-11-20T05:31:03.381 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_background_compactions: -1 2021-11-20T05:31:03.382 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-11-20T05:31:03.382 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-11-20T05:31:03.382 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-11-20T05:31:03.382 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_total_wal_size: 0 2021-11-20T05:31:03.383 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-11-20T05:31:03.383 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-11-20T05:31:03.383 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-11-20T05:31:03.383 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-11-20T05:31:03.383 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_open_files: -1 2021-11-20T05:31:03.384 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.bytes_per_sync: 0 2021-11-20T05:31:03.384 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-11-20T05:31:03.384 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-11-20T05:31:03.384 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compaction_readahead_size: 0 2021-11-20T05:31:03.385 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Compression algorithms supported: 2021-11-20T05:31:03.385 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-11-20T05:31:03.385 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: kZSTD supported: 0 2021-11-20T05:31:03.385 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: kXpressCompression supported: 0 2021-11-20T05:31:03.385 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: kLZ4HCCompression supported: 1 2021-11-20T05:31:03.386 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: kLZ4Compression supported: 1 2021-11-20T05:31:03.386 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: kBZip2Compression supported: 0 2021-11-20T05:31:03.386 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: kZlibCompression supported: 1 2021-11-20T05:31:03.386 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: kSnappyCompression supported: 1 2021-11-20T05:31:03.387 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-11-20T05:31:03.387 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi187/store.db/MANIFEST-000001 2021-11-20T05:31:03.387 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.387 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-11-20T05:31:03.388 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.388 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-11-20T05:31:03.388 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.merge_operator: 2021-11-20T05:31:03.388 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compaction_filter: None 2021-11-20T05:31:03.388 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compaction_filter_factory: None 2021-11-20T05:31:03.389 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.389 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-11-20T05:31:03.389 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-11-20T05:31:03.389 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557e3965c128) 2021-11-20T05:31:03.390 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: cache_index_and_filter_blocks: 1 2021-11-20T05:31:03.390 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: cache_index_and_filter_blocks_with_high_priority: 0 2021-11-20T05:31:03.390 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-11-20T05:31:03.390 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: pin_top_level_index_and_filter: 1 2021-11-20T05:31:03.390 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: index_type: 0 2021-11-20T05:31:03.391 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: data_block_index_type: 0 2021-11-20T05:31:03.391 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: index_shortening: 1 2021-11-20T05:31:03.391 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: data_block_hash_table_util_ratio: 0.750000 2021-11-20T05:31:03.391 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: hash_index_allow_collision: 1 2021-11-20T05:31:03.392 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: checksum: 1 2021-11-20T05:31:03.392 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: no_block_cache: 0 2021-11-20T05:31:03.392 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: block_cache: 0x557e39692f10 2021-11-20T05:31:03.392 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: block_cache_name: BinnedLRUCache 2021-11-20T05:31:03.392 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: block_cache_options: 2021-11-20T05:31:03.393 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: capacity : 536870912 2021-11-20T05:31:03.393 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: num_shard_bits : 4 2021-11-20T05:31:03.393 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: strict_capacity_limit : 0 2021-11-20T05:31:03.393 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: high_pri_pool_ratio: 0.000 2021-11-20T05:31:03.393 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: block_cache_compressed: (nil) 2021-11-20T05:31:03.394 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: persistent_cache: (nil) 2021-11-20T05:31:03.394 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: block_size: 4096 2021-11-20T05:31:03.394 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: block_size_deviation: 10 2021-11-20T05:31:03.394 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: block_restart_interval: 16 2021-11-20T05:31:03.395 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: index_block_restart_interval: 1 2021-11-20T05:31:03.395 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: metadata_block_size: 4096 2021-11-20T05:31:03.395 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: partition_filters: 0 2021-11-20T05:31:03.395 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: use_delta_encoding: 1 2021-11-20T05:31:03.395 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: filter_policy: rocksdb.BuiltinBloomFilter 2021-11-20T05:31:03.396 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: whole_key_filtering: 1 2021-11-20T05:31:03.396 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: verify_compression: 0 2021-11-20T05:31:03.396 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: read_amp_bytes_per_bit: 0 2021-11-20T05:31:03.396 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: format_version: 2 2021-11-20T05:31:03.397 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: enable_index_compression: 1 2021-11-20T05:31:03.397 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: block_align: 0 2021-11-20T05:31:03.397 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.397 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.write_buffer_size: 33554432 2021-11-20T05:31:03.398 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_write_buffer_number: 2 2021-11-20T05:31:03.398 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compression: NoCompression 2021-11-20T05:31:03.398 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.bottommost_compression: Disabled 2021-11-20T05:31:03.398 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.prefix_extractor: nullptr 2021-11-20T05:31:03.398 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-11-20T05:31:03.399 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.num_levels: 7 2021-11-20T05:31:03.399 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.399 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-11-20T05:31:03.399 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-11-20T05:31:03.399 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-11-20T05:31:03.400 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-11-20T05:31:03.400 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-11-20T05:31:03.400 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-11-20T05:31:03.400 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-11-20T05:31:03.401 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-11-20T05:31:03.401 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-11-20T05:31:03.401 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-11-20T05:31:03.401 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compression_opts.level: 32767 2021-11-20T05:31:03.401 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compression_opts.strategy: 0 2021-11-20T05:31:03.402 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-11-20T05:31:03.402 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-11-20T05:31:03.402 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compression_opts.enabled: false 2021-11-20T05:31:03.402 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-11-20T05:31:03.403 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-11-20T05:31:03.403 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-11-20T05:31:03.403 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.target_file_size_base: 67108864 2021-11-20T05:31:03.403 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-11-20T05:31:03.403 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-11-20T05:31:03.404 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-11-20T05:31:03.404 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-11-20T05:31:03.404 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-11-20T05:31:03.404 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-11-20T05:31:03.405 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-11-20T05:31:03.405 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-11-20T05:31:03.405 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-11-20T05:31:03.405 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-11-20T05:31:03.406 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-11-20T05:31:03.406 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-11-20T05:31:03.406 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-11-20T05:31:03.406 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.arena_block_size: 4194304 2021-11-20T05:31:03.406 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-11-20T05:31:03.407 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-11-20T05:31:03.407 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-11-20T05:31:03.407 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.disable_auto_compactions: 0 2021-11-20T05:31:03.407 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-11-20T05:31:03.408 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-11-20T05:31:03.408 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-11-20T05:31:03.408 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-11-20T05:31:03.408 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-11-20T05:31:03.408 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-11-20T05:31:03.409 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-11-20T05:31:03.409 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-11-20T05:31:03.409 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-11-20T05:31:03.409 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-11-20T05:31:03.409 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.table_properties_collectors: 2021-11-20T05:31:03.410 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.inplace_update_support: 0 2021-11-20T05:31:03.410 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-11-20T05:31:03.410 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-11-20T05:31:03.410 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-11-20T05:31:03.411 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-11-20T05:31:03.411 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.bloom_locality: 0 2021-11-20T05:31:03.411 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.max_successive_merges: 0 2021-11-20T05:31:03.411 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-11-20T05:31:03.412 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.paranoid_file_checks: 0 2021-11-20T05:31:03.412 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.force_consistency_checks: 0 2021-11-20T05:31:03.412 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.report_bg_io_stats: 0 2021-11-20T05:31:03.412 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.ttl: 2592000 2021-11-20T05:31:03.412 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-11-20T05:31:03.413 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi187/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-11-20T05:31:03.413 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.413 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-11-20T05:31:03.413 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.414 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637386263057260, "job": 1, "event": "recovery_started", "log_files": [3]} 2021-11-20T05:31:03.414 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2021-11-20T05:31:03.414 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.056+0000 7f4f64aba700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-11-20T05:31:03.414 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.057+0000 7f4f64aba700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637386263057733, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637386263, "oldest_key_time": 3, "file_creation_time": 0}} 2021-11-20T05:31:03.414 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.057+0000 7f4f64aba700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2021-11-20T05:31:03.415 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.415 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.057+0000 7f4f64aba700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637386263058102, "job": 1, "event": "recovery_finished"} 2021-11-20T05:31:03.415 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.058+0000 7f4f64aba700 4 rocksdb: DB pointer 0x557e39757800 2021-11-20T05:31:03.415 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.058+0000 7f4f4b8d2700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-11-20T05:31:03.416 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.058+0000 7f4f4b8d2700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-11-20T05:31:03.416 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: ** DB Stats ** 2021-11-20T05:31:03.416 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T05:31:03.416 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-11-20T05:31:03.416 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-11-20T05:31:03.417 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T05:31:03.417 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-11-20T05:31:03.417 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-11-20T05:31:03.417 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T05:31:03.417 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.418 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: ** Compaction Stats [default] ** 2021-11-20T05:31:03.418 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:31:03.418 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:31:03.418 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2021-11-20T05:31:03.419 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2021-11-20T05:31:03.419 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2021-11-20T05:31:03.419 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.419 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: ** Compaction Stats [default] ** 2021-11-20T05:31:03.419 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:31:03.420 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:31:03.420 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2021-11-20T05:31:03.420 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T05:31:03.420 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T05:31:03.421 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T05:31:03.421 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T05:31:03.421 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T05:31:03.421 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: AddFile(Keys): cumulative 0, interval 0 2021-11-20T05:31:03.421 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Cumulative compaction: 0.00 GB write, 0.53 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:31:03.422 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Interval compaction: 0.00 GB write, 0.53 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:31:03.422 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T05:31:03.422 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.422 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: ** File Read Latency Histogram By Level [default] ** 2021-11-20T05:31:03.423 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.423 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: ** Compaction Stats [default] ** 2021-11-20T05:31:03.423 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:31:03.423 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:31:03.424 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2021-11-20T05:31:03.424 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2021-11-20T05:31:03.424 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T05:31:03.424 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.424 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: ** Compaction Stats [default] ** 2021-11-20T05:31:03.425 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:31:03.425 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:31:03.425 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2021-11-20T05:31:03.425 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T05:31:03.426 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T05:31:03.426 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T05:31:03.426 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T05:31:03.426 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T05:31:03.426 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: AddFile(Keys): cumulative 0, interval 0 2021-11-20T05:31:03.427 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Cumulative compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:31:03.427 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:31:03.427 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T05:31:03.427 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.428 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: ** File Read Latency Histogram By Level [default] ** 2021-11-20T05:31:03.428 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.428 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.058+0000 7f4f64aba700 0 mon.smithi187 does not exist in monmap, will attempt to join an existing cluster 2021-11-20T05:31:03.428 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.059+0000 7f4f64aba700 0 using public_addr v2:172.21.15.187:0/0 -> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] 2021-11-20T05:31:03.428 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.059+0000 7f4f64aba700 0 starting mon.smithi187 rank -1 at public addrs [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] at bind addrs [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi187 fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:31:03.429 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.060+0000 7f4f64aba700 1 mon.smithi187@-1(???) e0 preinit fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:31:03.429 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.089+0000 7f4f4e0d7700 0 mon.smithi187@-1(synchronizing).mds e1 new map 2021-11-20T05:31:03.429 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.089+0000 7f4f4e0d7700 0 mon.smithi187@-1(synchronizing).mds e1 print_map 2021-11-20T05:31:03.430 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: e1 2021-11-20T05:31:03.430 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: enable_multiple, ever_enabled_multiple: 1,1 2021-11-20T05:31:03.430 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T05:31:03.430 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: legacy client fscid: -1 2021-11-20T05:31:03.430 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.431 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: No filesystems configured 2021-11-20T05:31:03.431 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.431 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.089+0000 7f4f4e0d7700 1 mon.smithi187@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2021-11-20T05:31:03.431 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.089+0000 7f4f4e0d7700 1 mon.smithi187@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2021-11-20T05:31:03.432 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.089+0000 7f4f4e0d7700 1 mon.smithi187@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2021-11-20T05:31:03.432 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.089+0000 7f4f4e0d7700 1 mon.smithi187@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2021-11-20T05:31:03.432 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.089+0000 7f4f4e0d7700 1 mon.smithi187@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2021-11-20T05:31:03.432 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.089+0000 7f4f4e0d7700 1 mon.smithi187@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2021-11-20T05:31:03.432 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.089+0000 7f4f4e0d7700 1 mon.smithi187@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2021-11-20T05:31:03.433 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.091+0000 7f4f4e0d7700 0 mon.smithi187@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2021-11-20T05:31:03.433 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.091+0000 7f4f4e0d7700 0 mon.smithi187@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T05:31:03.433 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.091+0000 7f4f4e0d7700 0 mon.smithi187@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T05:31:03.433 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.091+0000 7f4f4e0d7700 0 mon.smithi187@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T05:31:03.434 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: audit 2021-11-20T05:31:03.434 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:30:59 2021-11-20T05:31:03.434 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: .794090+0000 mon.smithi058 (mon.0) 155 2021-11-20T05:31:03.434 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:03.434 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: audit 2021-11-20T 2021-11-20T05:31:03.435 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 05:30:59.795423+0000 mon.smithi058 (mon. 2021-11-20T05:31:03.438 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 0) 156 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T05:31:03.438 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: audit 2021-11- 2021-11-20T05:31:03.438 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 20T05:30:59.796373+0000 mon.smithi058 2021-11-20T05:31:03.439 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: (mon.0) 157 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:03.439 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:03.439 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: cephadm 2021-11-20T 2021-11-20T05:31:03.440 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 05:30:59.797195+0000 mgr.smithi058.zagdlk (mgr.14164) 38 : 2021-11-20T05:31:03.441 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: cephadm [INF] Deploying daemon mon.smithi187 on smithi187 2021-11-20T05:31:03.441 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: cluster 2021 2021-11-20T05:31:03.441 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: -11-20T05:31:00.177291+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:31:03.441 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 14164) 39 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:03.442 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: audit 2021-11-20T05:31 2021-11-20T05:31:03.442 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: :00.355074+0000 mon.smithi058 (mon.0) 158 2021-11-20T05:31:03.442 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:31:03.443 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: audit 2021-11-20T05:31:03.443 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: 2021-11-20T05:31:00.362714+0000 mon.smithi058 (mon 2021-11-20T05:31:03.443 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: .0) 159 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:31:03.443 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 conmon[36358]: debug 2021-11-20T05:31:03.091+0000 7f4f4e0d7700 1 mon.smithi187@-1(synchronizing).paxosservice(auth 1..7) refresh upgraded, format 0 -> 3 2021-11-20T05:31:03.443 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:03 smithi187 systemd[1]: Started Ceph mon.smithi187 for b33f8c4a-49c2-11ec-8c2c-001a4aab830c. 2021-11-20T05:31:04.375 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T05:31:04.375 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T05:31:08.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: audit 2021-11-20T05:31:03.098936+0000 mon.smithi058 (mon.0) 162 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi058"}]: dispatch 2021-11-20T05:31:08.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: cluster 2021-11-20T05:31:03.099134+0000 mon.smithi058 (mon.0) 163 : cluster [INF] mon.smithi058 calling monitor election 2021-11-20T05:31:08.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: audit 2021-11-20T05:31:03.112956+0000 mon.smithi058 (mon.0) 164 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2021-11-20T05:31:08.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: audit 2021-11-20T05:31:04.096081+0000 mon.smithi058 (mon.0) 165 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2021-11-20T05:31:08.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: cluster 2021-11-20T05:31:04.177951+0000 mgr.smithi058.zagdlk (mgr.14164) 41 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:08.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: audit 2021-11-20T05:31:04.857667+0000 mon.smithi058 (mon.0) 166 : audit [DBG] from='mgr.? 172.21.15.187:0/3190706472' entity='mgr.smithi187.omkctn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi187.omkctn/crt"}]: dispatch 2021-11-20T05:31:08.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: audit 2021-11-20T05:31:05.096535+0000 mon.smithi058 (mon.0) 167 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2021-11-20T05:31:08.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: cluster 2021-11-20T05:31:05.097044+0000 mon.smithi187 (mon.1) 1 : cluster [INF] mon.smithi187 calling monitor election 2021-11-20T05:31:08.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: audit 2021-11-20T05:31:06.096081+0000 mon.smithi058 (mon.0) 168 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2021-11-20T05:31:08.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: cluster 2021-11-20T05:31:06.178232+0000 mgr.smithi058.zagdlk (mgr.14164) 42 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:08.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: audit 2021-11-20T05:31:07.096160+0000 mon.smithi058 (mon.0) 169 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2021-11-20T05:31:08.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: audit 2021-11-20T05:31:08.096323+0000 mon.smithi058 (mon.0) 170 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2021-11-20T05:31:08.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: cluster 2021-11-20T05:31:08.126769+0000 mon.smithi058 (mon.0) 171 : cluster [INF] mon.smithi058 is new leader, mons smithi058,smithi187 in quorum (ranks 0,1) 2021-11-20T05:31:08.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: cluster 2021-11-20T05:31:08.134841+0000 mon.smithi058 (mon.0) 172 : cluster [DBG] monmap e2: 2 mons at {smithi058=[v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0],smithi187=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0]} 2021-11-20T05:31:08.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: cluster 2021-11-20T05:31:08.134912+0000 mon.smithi058 (mon.0) 173 : cluster [DBG] fsmap 2021-11-20T05:31:08.451 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: cluster 2021-11-20T05:31:08.134944+0000 mon.smithi058 (mon.0) 174 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2021-11-20T05:31:08.451 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: cluster 2021-11-20T05:31:08.135169+0000 mon.smithi058 (mon.0) 175 : cluster [DBG] mgrmap e17: smithi058.zagdlk(active, since 68s) 2021-11-20T05:31:08.451 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: cluster 2021-11-20T05:31:08.136180+0000 mon.smithi058 (mon.0) 176 : cluster [DBG] Standby manager daemon smithi187.omkctn started 2021-11-20T05:31:08.451 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: audit 2021-11-20T05:31:08.137254+0000 mon.smithi058 (mon.0) 177 : audit [DBG] from='mgr.? 172.21.15.187:0/3190706472' entity='mgr.smithi187.omkctn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-11-20T05:31:08.451 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: audit 2021-11-20T05:31:08.138695+0000 mon.smithi058 (mon.0) 178 : audit [DBG] from='mgr.? 172.21.15.187:0/3190706472' entity='mgr.smithi187.omkctn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi187.omkctn/key"}]: dispatch 2021-11-20T05:31:08.452 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: audit 2021-11-20T05:31:08.139231+0000 mon.smithi058 (mon.0) 179 : audit [DBG] from='mgr.? 172.21.15.187:0/3190706472' entity='mgr.smithi187.omkctn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-11-20T05:31:08.452 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: cluster 2021-11-20T05:31:08.142508+0000 mon.smithi058 (mon.0) 180 : cluster [INF] overall HEALTH_OK 2021-11-20T05:31:08.452 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:08 smithi187 conmon[36358]: audit 2021-11-20T05:31:08.146609+0000 mon.smithi058 (mon.0) 181 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:08.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: audit 2021-11-20T05:31:03.098936+0000 mon.smithi058 ( 2021-11-20T05:31:08.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: mon.0) 162 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi058"}]: dispatch 2021-11-20T05:31:08.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: cluster 2021-11-20T05:31:03.099134+0000 2021-11-20T05:31:08.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: mon.smithi058 (mon.0) 163 : cluster [INF] mon.smithi058 calling monitor election 2021-11-20T05:31:08.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:31:08.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 11-20T05:31:03 2021-11-20T05:31:08.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: .112956+0000 mon.smithi058 (mon.0) 2021-11-20T05:31:08.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 164 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2021-11-20T05:31:08.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: audit 2021-11-20T05:31:08.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 2021-11- 2021-11-20T05:31:08.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 20T05:31:04.096081+0000 2021-11-20T05:31:08.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: mon.smithi058 (mon.0) 165 : audit [DBG] 2021-11-20T05:31:08.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2021-11-20T05:31:08.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: cluster 2021- 2021-11-20T05:31:08.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 11-20T05:31:04 2021-11-20T05:31:08.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: .177951+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:31:08.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: ) 41 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:08.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:31:08.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: -20T05:31:04.857667+0000 2021-11-20T05:31:08.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: mon.smithi058 (mon.0) 166 : audit [DBG] 2021-11-20T05:31:08.475 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: from='mgr.? 172.21.15.187:0/3190706472' entity='mgr.smithi187.omkctn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi187.omkctn/crt"}]: dispatch 2021-11-20T05:31:08.475 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:31:08.475 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: -20T05:31:05. 2021-11-20T05:31:08.476 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 096535+0000 mon.smithi058 (mon.0) 167 : 2021-11-20T05:31:08.476 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2021-11-20T05:31:08.476 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:31:08.476 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: -11-20T05:31: 2021-11-20T05:31:08.476 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 05.097044+0000 mon.smithi187 (mon.1) 2021-11-20T05:31:08.477 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 1 : cluster [INF] mon.smithi187 calling monitor election 2021-11-20T05:31:08.477 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: audit 2021-11-20T05:31:08.477 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 2021-11-20T05:31 2021-11-20T05:31:08.478 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: :06.096081+0000 mon.smithi058 (mon. 2021-11-20T05:31:08.478 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 0) 168 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2021-11-20T05:31:08.478 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: cluster 2021-11-20T05:31:08.478 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 2021-11-20T05: 2021-11-20T05:31:08.479 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 31:06.178232+0000 mgr.smithi058.zagdlk (mgr 2021-11-20T05:31:08.479 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: .14164) 42 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:08.479 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:31:08.479 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 11-20T05:31:07. 2021-11-20T05:31:08.480 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 096160+0000 mon.smithi058 (mon.0) 169 : 2021-11-20T05:31:08.480 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2021-11-20T05:31:08.480 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: audit 2021 2021-11-20T05:31:08.480 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: -11-20T05:31: 2021-11-20T05:31:08.481 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 08.096323+0000 mon.smithi058 (mon.0) 2021-11-20T05:31:08.481 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 170 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2021-11-20T05:31:08.481 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:31:08.481 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: -11-20T05:31 2021-11-20T05:31:08.482 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: :08.126769+0000 mon.smithi058 (mon.0 2021-11-20T05:31:08.482 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: ) 171 : cluster [INF] mon.smithi058 is new leader, mons smithi058,smithi187 in quorum (ranks 0,1) 2021-11-20T05:31:08.482 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: cluster 2021-11-20T05:31:08.482 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 2021-11-20T05:31 2021-11-20T05:31:08.483 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: :08.134841+0000 mon.smithi058 (mon. 2021-11-20T05:31:08.483 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 0) 172 : cluster [DBG] monmap e2: 2 mons at {smithi058=[v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0],smithi187=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0]} 2021-11-20T05:31:08.483 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: cluster 2021-11-20T05:31:08.483 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 2021-11-20T05:31 2021-11-20T05:31:08.483 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: :08.134912+0000 mon.smithi058 (mon.0 2021-11-20T05:31:08.484 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: ) 173 : cluster [DBG] fsmap 2021-11-20T05:31:08.484 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: cluster 2021-11-20T05:31:08.484 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 2021-11-20T05:31:08 2021-11-20T05:31:08.484 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: .134944+0000 mon.smithi058 (mon.0) 174 : cluster 2021-11-20T05:31:08.485 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: [DBG] osdmap e5: 0 total, 0 up, 0 in 2021-11-20T05:31:08.485 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: cluster 2021-11-20T05:31:08.485 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 2021-11-20T05:31 2021-11-20T05:31:08.485 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: :08.135169+0000 mon.smithi058 (mon. 2021-11-20T05:31:08.485 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 0) 175 : cluster [DBG] mgrmap e17: smithi058.zagdlk(active, since 68s) 2021-11-20T05:31:08.486 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: cluster 2021-11-20T05:31:08.486 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 2021-11-20T05: 2021-11-20T05:31:08.486 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 31:08.136180+0000 mon.smithi058 (mon. 2021-11-20T05:31:08.486 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 0) 176 : cluster [DBG] Standby manager daemon smithi187.omkctn started 2021-11-20T05:31:08.487 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: audit 2021-11-20T05:31:08.487 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 2021-11-20T05: 2021-11-20T05:31:08.487 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 31:08.137254+0000 mon.smithi058 2021-11-20T05:31:08.487 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: (mon.0) 177 : audit 2021-11-20T05:31:08.488 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: [DBG] from='mgr.? 172.21.15.187:0/3190706472' entity='mgr.smithi187.omkctn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-11-20T05:31:08.488 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: audit 2021 2021-11-20T05:31:08.488 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: -11-20T05:31: 2021-11-20T05:31:08.488 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 08.138695+0000 mon.smithi058 (mon.0) 2021-11-20T05:31:08.489 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 178 : audit [DBG] from='mgr.? 172.21.15.187:0/3190706472' entity='mgr.smithi187.omkctn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi187.omkctn/key"}]: dispatch 2021-11-20T05:31:08.489 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: audit 2021-11-20T05:31:08.489 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 2021-11-20T05:31 2021-11-20T05:31:08.489 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: :08.139231+0000 mon.smithi058 (mon. 2021-11-20T05:31:08.489 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 0) 179 : audit [DBG] from='mgr.? 172.21.15.187:0/3190706472' entity='mgr.smithi187.omkctn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-11-20T05:31:08.490 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: cluster 2021-11-20T05:31:08.490 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 2021-11-20T05:31 2021-11-20T05:31:08.490 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: :08.142508+0000 mon.smithi058 (mon 2021-11-20T05:31:08.490 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: .0) 180 : cluster [INF] overall HEALTH_OK 2021-11-20T05:31:08.491 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: audit 2021-11-20T05:31:08.491 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 2021-11-20T05: 2021-11-20T05:31:08.491 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 31:08.146609+0000 mon.smithi058 (mon. 2021-11-20T05:31:08.491 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:08 smithi058 conmon[33537]: 0) 181 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:09.083 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.081+0000 7fecc8a35700 1 -- 172.21.15.187:0/841821876 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fecac003250 msgr2=0x7fecac0056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:09.084 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.081+0000 7fecc8a35700 1 --2- 172.21.15.187:0/841821876 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fecac003250 0x7fecac0056e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fecc40f5770 tx=0x7fecb8002040).stop 2021-11-20T05:31:09.084 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.081+0000 7fecc8a35700 1 -- 172.21.15.187:0/841821876 shutdown_connections 2021-11-20T05:31:09.084 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.081+0000 7fecc8a35700 1 --2- 172.21.15.187:0/841821876 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fecac003250 0x7fecac0056e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:09.085 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.081+0000 7fecc8a35700 1 -- 172.21.15.187:0/841821876 >> 172.21.15.187:0/841821876 conn(0x7fecc40f0120 msgr2=0x7fecc40f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:31:09.085 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.081+0000 7fecc8a35700 1 -- 172.21.15.187:0/841821876 shutdown_connections 2021-11-20T05:31:09.085 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.081+0000 7fecc8a35700 1 -- 172.21.15.187:0/841821876 wait complete. 2021-11-20T05:31:09.085 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.082+0000 7fecc8a35700 1 Processor -- start 2021-11-20T05:31:09.086 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.082+0000 7fecc8a35700 1 -- start start 2021-11-20T05:31:09.086 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.082+0000 7fecc8a35700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fecac003250 0x7fecc4106490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:31:09.086 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.082+0000 7fecc8a35700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc4106980 0x7fecc4107870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:31:09.086 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.082+0000 7fecc8a35700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc4106ee0 con 0x7fecac003250 2021-11-20T05:31:09.087 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.082+0000 7fecc8a35700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc4107020 con 0x7fecc4106980 2021-11-20T05:31:09.087 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.082+0000 7fecc2ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc4106980 0x7fecc4107870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:31:09.087 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.082+0000 7fecc2ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc4106980 0x7fecc4107870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44318/0 (socket says 172.21.15.187:44318) 2021-11-20T05:31:09.087 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.082+0000 7fecc2ffd700 1 -- 172.21.15.187:0/2866024516 learned_addr learned my addr 172.21.15.187:0/2866024516 (peer_addr_for_me v2:172.21.15.187:0/0) 2021-11-20T05:31:09.088 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.082+0000 7fecc37fe700 1 --2- 172.21.15.187:0/2866024516 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fecac003250 0x7fecc4106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:31:09.088 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.082+0000 7fecc2ffd700 1 -- 172.21.15.187:0/2866024516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc4106980 msgr2=0x7fecc4107870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-11-20T05:31:09.088 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.082+0000 7fecc2ffd700 1 -- 172.21.15.187:0/2866024516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc4106980 msgr2=0x7fecc4107870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-11-20T05:31:09.088 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.082+0000 7fecc2ffd700 1 --2- 172.21.15.187:0/2866024516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc4106980 0x7fecc4107870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T05:31:09.088 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.083+0000 7fecc2ffd700 1 --2- 172.21.15.187:0/2866024516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc4106980 0x7fecc4107870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T05:31:09.089 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.083+0000 7fecc37fe700 1 -- 172.21.15.187:0/2866024516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc4106980 msgr2=0x7fecc4107870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T05:31:09.089 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.083+0000 7fecc37fe700 1 --2- 172.21.15.187:0/2866024516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc4106980 0x7fecc4107870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:09.089 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.083+0000 7fecc37fe700 1 -- 172.21.15.187:0/2866024516 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb8011040 con 0x7fecac003250 2021-11-20T05:31:09.089 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.083+0000 7fecc37fe700 1 --2- 172.21.15.187:0/2866024516 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fecac003250 0x7fecc4106490 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fecb8000c00 tx=0x7fecb8002040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:31:09.090 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.083+0000 7fecabfff700 1 -- 172.21.15.187:0/2866024516 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecb800e040 con 0x7fecac003250 2021-11-20T05:31:09.090 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.083+0000 7fecabfff700 1 -- 172.21.15.187:0/2866024516 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecb8018450 con 0x7fecac003250 2021-11-20T05:31:09.090 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.083+0000 7fecc8a35700 1 -- 172.21.15.187:0/2866024516 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc4107e20 con 0x7fecac003250 2021-11-20T05:31:09.090 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.083+0000 7fecabfff700 1 -- 172.21.15.187:0/2866024516 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecb8021690 con 0x7fecac003250 2021-11-20T05:31:09.090 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.083+0000 7fecc8a35700 1 -- 172.21.15.187:0/2866024516 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc41083e0 con 0x7fecac003250 2021-11-20T05:31:09.092 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.084+0000 7fecc8a35700 1 -- 172.21.15.187:0/2866024516 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc4045ad0 con 0x7fecac003250 2021-11-20T05:31:09.092 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.085+0000 7fecabfff700 1 -- 172.21.15.187:0/2866024516 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fecb8028030 con 0x7fecac003250 2021-11-20T05:31:09.092 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.085+0000 7fecabfff700 1 --2- 172.21.15.187:0/2866024516 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fecb005c3b0 0x7fecb005e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:31:09.092 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.085+0000 7fecabfff700 1 -- 172.21.15.187:0/2866024516 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fecb80766e0 con 0x7fecac003250 2021-11-20T05:31:09.093 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.085+0000 7fecc2ffd700 1 --2- 172.21.15.187:0/2866024516 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fecb005c3b0 0x7fecb005e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:31:09.093 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.085+0000 7fecc2ffd700 1 --2- 172.21.15.187:0/2866024516 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fecb005c3b0 0x7fecb005e870 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fecb4006cf0 tx=0x7fecb4008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:31:09.093 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.090+0000 7fecabfff700 1 -- 172.21.15.187:0/2866024516 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fecb804a460 con 0x7fecac003250 2021-11-20T05:31:09.304 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.304+0000 7fecc8a35700 1 -- 172.21.15.187:0/2866024516 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fecc4108650 con 0x7fecac003250 2021-11-20T05:31:09.305 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:09 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:08.147689+0000 mgr.smithi058.zagdlk (mgr.14164) 43 : cephadm [INF] Deploying daemon node-exporter.smithi187 on smithi187 2021-11-20T05:31:09.305 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:09 smithi187 conmon[36358]: cluster 2021-11-20T05:31:08.178597+0000 mgr.smithi058.zagdlk (mgr.14164) 44 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:09.305 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:09 smithi187 conmon[36358]: cluster 2021-11-20T05:31:08.195180+0000 mon.smithi058 (mon.0) 182 : cluster [DBG] mgrmap e18: smithi058.zagdlk(active, since 68s), standbys: smithi187.omkctn 2021-11-20T05:31:09.305 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:09 smithi187 conmon[36358]: audit 2021-11-20T05:31:08.195335+0000 mon.smithi058 (mon.0) 183 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mgr metadata", "who": "smithi187.omkctn", "id": "smithi187.omkctn"}]: dispatch 2021-11-20T05:31:09.306 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:09 smithi187 conmon[36358]: audit 2021-11-20T05:31:09.096163+0000 mon.smithi058 (mon.0) 184 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2021-11-20T05:31:09.306 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.304+0000 7fecabfff700 1 -- 172.21.15.187:0/2866024516 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+968 (secure 0 0 0) 0x7fecb801d070 con 0x7fecac003250 2021-11-20T05:31:09.307 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-11-20T05:31:09.307 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":2,"fsid":"b33f8c4a-49c2-11ec-8c2c-001a4aab830c","modified":"2021-11-20T05:31:03.096054Z","created":"2021-11-20T05:28:52.613926Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi058","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:3300","nonce":0},{"type":"v1","addr":"172.21.15.58:6789","nonce":0}]},"addr":"172.21.15.58:6789/0","public_addr":"172.21.15.58:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi187","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:3300","nonce":0},{"type":"v1","addr":"172.21.15.187:6789","nonce":0}]},"addr":"172.21.15.187:6789/0","public_addr":"172.21.15.187:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2021-11-20T05:31:09.308 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.306+0000 7fecc8a35700 1 -- 172.21.15.187:0/2866024516 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fecb005c3b0 msgr2=0x7fecb005e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:09.308 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.306+0000 7fecc8a35700 1 --2- 172.21.15.187:0/2866024516 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fecb005c3b0 0x7fecb005e870 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fecb4006cf0 tx=0x7fecb4008040).stop 2021-11-20T05:31:09.309 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.306+0000 7fecc8a35700 1 -- 172.21.15.187:0/2866024516 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fecac003250 msgr2=0x7fecc4106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:09.309 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.306+0000 7fecc8a35700 1 --2- 172.21.15.187:0/2866024516 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fecac003250 0x7fecc4106490 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fecb8000c00 tx=0x7fecb8002040).stop 2021-11-20T05:31:09.309 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.306+0000 7fecc8a35700 1 -- 172.21.15.187:0/2866024516 shutdown_connections 2021-11-20T05:31:09.309 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.306+0000 7fecc8a35700 1 --2- 172.21.15.187:0/2866024516 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fecb005c3b0 0x7fecb005e870 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:09.310 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.306+0000 7fecc8a35700 1 --2- 172.21.15.187:0/2866024516 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fecac003250 0x7fecc4106490 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:09.310 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.306+0000 7fecc8a35700 1 --2- 172.21.15.187:0/2866024516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc4106980 0x7fecc4107870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:09.310 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.306+0000 7fecc8a35700 1 -- 172.21.15.187:0/2866024516 >> 172.21.15.187:0/2866024516 conn(0x7fecc40f0120 msgr2=0x7fecc40fab80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:31:09.310 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.307+0000 7fecc8a35700 1 -- 172.21.15.187:0/2866024516 shutdown_connections 2021-11-20T05:31:09.311 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:31:09.307+0000 7fecc8a35700 1 -- 172.21.15.187:0/2866024516 wait complete. 2021-11-20T05:31:09.311 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 2 2021-11-20T05:31:09.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:09 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:08.147689+0000 mgr.smithi058.zagdlk (mgr.14164) 43 : cephadm [INF] Deploying daemon node-exporter.smithi187 on smithi187 2021-11-20T05:31:09.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:09 smithi058 conmon[33537]: cluster 2021-11-20T05:31:08.178597+0000 mgr.smithi058.zagdlk (mgr.14164) 44 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:09.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:09 smithi058 conmon[33537]: cluster 2021-11-20T 2021-11-20T05:31:09.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:09 smithi058 conmon[33537]: 05:31:08.195180+0000 mon.smithi058 (mon.0) 182 : cluster [DBG] 2021-11-20T05:31:09.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:09 smithi058 conmon[33537]: mgrmap e18: smithi058.zagdlk(active, since 68s), standbys: smithi187.omkctn 2021-11-20T05:31:09.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:09 smithi058 conmon[33537]: audit 2021 2021-11-20T05:31:09.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:09 smithi058 conmon[33537]: -11-20T05:31: 2021-11-20T05:31:09.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:09 smithi058 conmon[33537]: 08.195335+0000 mon.smithi058 (mon.0 2021-11-20T05:31:09.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:09 smithi058 conmon[33537]: ) 183 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mgr metadata", "who": "smithi187.omkctn", "id": "smithi187.omkctn"}]: dispatch 2021-11-20T05:31:09.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:09 smithi058 conmon[33537]: audit 2021-11-20T05:31:09.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:09 smithi058 conmon[33537]: 2021-11 2021-11-20T05:31:09.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:09 smithi058 conmon[33537]: -20T05:31:09.096163 2021-11-20T05:31:09.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:09 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 184 : audit 2021-11-20T05:31:09.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:09 smithi058 conmon[33537]: [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2021-11-20T05:31:10.164 INFO:tasks.cephadm:Generating final ceph.conf file... 2021-11-20T05:31:10.164 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph config generate-minimal-conf 2021-11-20T05:31:10.395 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:10 smithi187 conmon[36358]: audit 2021-11-20T05:31:09.306120+0000 2021-11-20T05:31:10.395 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:10 smithi187 conmon[36358]: mon.smithi058 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.187:0/2866024516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T05:31:10.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:10 smithi058 conmon[33537]: audit 2021-11-20T05:31:09.306120+0000 mon.smithi058 (mon.0) 185 2021-11-20T05:31:10.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:10 smithi058 conmon[33537]: : audit [DBG] from='client.? 172.21.15.187:0/2866024516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T05:31:11.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:11 smithi187 conmon[36358]: cluster 2021-11-20T05:31:10.178904+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:31:11.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:11 smithi187 conmon[36358]: mgr.14164) 45 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:11.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:11 smithi058 conmon[33537]: cluster 2021-11-20T05:31:10.178904+0000 mgr.smithi058.zagdlk (mgr.14164) 45 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:11.960 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.958+0000 7fa50e332700 1 -- 172.21.15.58:0/2151209547 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa5080f5410 msgr2=0x7fa5080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:11.961 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.958+0000 7fa50e332700 1 --2- 172.21.15.58:0/2151209547 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa5080f5410 0x7fa5080f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fa4f8009fb0 tx=0x7fa4f8009b30).stop 2021-11-20T05:31:11.961 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.958+0000 7fa50e332700 1 -- 172.21.15.58:0/2151209547 shutdown_connections 2021-11-20T05:31:11.962 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.958+0000 7fa50e332700 1 --2- 172.21.15.58:0/2151209547 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa5080f5410 0x7fa5080f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:11.962 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.958+0000 7fa50e332700 1 -- 172.21.15.58:0/2151209547 >> 172.21.15.58:0/2151209547 conn(0x7fa5080f0b70 msgr2=0x7fa5080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:31:11.962 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.958+0000 7fa50e332700 1 -- 172.21.15.58:0/2151209547 shutdown_connections 2021-11-20T05:31:11.962 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.958+0000 7fa50e332700 1 -- 172.21.15.58:0/2151209547 wait complete. 2021-11-20T05:31:11.963 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.959+0000 7fa50e332700 1 Processor -- start 2021-11-20T05:31:11.963 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.959+0000 7fa50e332700 1 -- start start 2021-11-20T05:31:11.963 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.959+0000 7fa50e332700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa5080f5410 0x7fa5081094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:31:11.963 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.959+0000 7fa50e332700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa508060220 0x7fa50810b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:31:11.963 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.959+0000 7fa50e332700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50810bf90 con 0x7fa5080f5410 2021-11-20T05:31:11.964 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.959+0000 7fa50e332700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50810c0d0 con 0x7fa508060220 2021-11-20T05:31:11.964 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.959+0000 7fa507fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa5080f5410 0x7fa5081094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:31:11.964 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.959+0000 7fa507fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa5080f5410 0x7fa5081094c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39958/0 (socket says 172.21.15.58:39958) 2021-11-20T05:31:11.964 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.959+0000 7fa507fff700 1 -- 172.21.15.58:0/1247106512 learned_addr learned my addr 172.21.15.58:0/1247106512 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:31:11.965 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.960+0000 7fa507fff700 1 -- 172.21.15.58:0/1247106512 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa508060220 msgr2=0x7fa50810b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:11.965 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.960+0000 7fa507fff700 1 --2- 172.21.15.58:0/1247106512 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa508060220 0x7fa50810b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:31:11.965 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.960+0000 7fa507fff700 1 -- 172.21.15.58:0/1247106512 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f8005040 con 0x7fa5080f5410 2021-11-20T05:31:11.965 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.960+0000 7fa507fff700 1 --2- 172.21.15.58:0/1247106512 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa5080f5410 0x7fa5081094c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fa4f8002040 tx=0x7fa4f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:31:11.965 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.960+0000 7fa504ff9700 1 -- 172.21.15.58:0/1247106512 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4f8019070 con 0x7fa5080f5410 2021-11-20T05:31:11.966 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.960+0000 7fa50e332700 1 -- 172.21.15.58:0/1247106512 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa50810c350 con 0x7fa5080f5410 2021-11-20T05:31:11.966 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.960+0000 7fa504ff9700 1 -- 172.21.15.58:0/1247106512 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4f8007c70 con 0x7fa5080f5410 2021-11-20T05:31:11.966 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.960+0000 7fa50e332700 1 -- 172.21.15.58:0/1247106512 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa50810c8b0 con 0x7fa5080f5410 2021-11-20T05:31:11.966 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.960+0000 7fa504ff9700 1 -- 172.21.15.58:0/1247106512 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4f801d620 con 0x7fa5080f5410 2021-11-20T05:31:11.967 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.961+0000 7fa50e332700 1 -- 172.21.15.58:0/1247106512 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa508045ad0 con 0x7fa5080f5410 2021-11-20T05:31:11.968 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.961+0000 7fa504ff9700 1 -- 172.21.15.58:0/1247106512 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa4f8025070 con 0x7fa5080f5410 2021-11-20T05:31:11.968 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.962+0000 7fa504ff9700 1 --2- 172.21.15.58:0/1247106512 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa4f005c3b0 0x7fa4f005e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:31:11.968 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.962+0000 7fa504ff9700 1 -- 172.21.15.58:0/1247106512 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa4f80762b0 con 0x7fa5080f5410 2021-11-20T05:31:11.968 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.962+0000 7fa5077fe700 1 --2- 172.21.15.58:0/1247106512 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa4f005c3b0 0x7fa4f005e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:31:11.969 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.962+0000 7fa5077fe700 1 --2- 172.21.15.58:0/1247106512 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa4f005c3b0 0x7fa4f005e870 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fa50805f9a0 tx=0x7fa4fc006ae0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:31:11.969 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:11.966+0000 7fa504ff9700 1 -- 172.21.15.58:0/1247106512 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa4f8049f80 con 0x7fa5080f5410 2021-11-20T05:31:12.128 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:12.128+0000 7fa50e332700 1 -- 172.21.15.58:0/1247106512 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fa50810d170 con 0x7fa5080f5410 2021-11-20T05:31:12.128 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:12.128+0000 7fa504ff9700 1 -- 172.21.15.58:0/1247106512 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v12) v1 ==== 76+0+225 (secure 0 0 0) 0x7fa4f804d5a0 con 0x7fa5080f5410 2021-11-20T05:31:12.129 INFO:teuthology.orchestra.run.smithi058.stdout:# minimal ceph.conf for b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:31:12.129 INFO:teuthology.orchestra.run.smithi058.stdout:[global] 2021-11-20T05:31:12.129 INFO:teuthology.orchestra.run.smithi058.stdout: fsid = b33f8c4a-49c2-11ec-8c2c-001a4aab830c 2021-11-20T05:31:12.129 INFO:teuthology.orchestra.run.smithi058.stdout: mon_host = [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] 2021-11-20T05:31:12.130 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:12.130+0000 7fa50e332700 1 -- 172.21.15.58:0/1247106512 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa4f005c3b0 msgr2=0x7fa4f005e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:12.131 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:12.130+0000 7fa50e332700 1 --2- 172.21.15.58:0/1247106512 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa4f005c3b0 0x7fa4f005e870 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fa50805f9a0 tx=0x7fa4fc006ae0).stop 2021-11-20T05:31:12.131 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:12.130+0000 7fa50e332700 1 -- 172.21.15.58:0/1247106512 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa5080f5410 msgr2=0x7fa5081094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:12.131 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:12.130+0000 7fa50e332700 1 --2- 172.21.15.58:0/1247106512 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa5080f5410 0x7fa5081094c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fa4f8002040 tx=0x7fa4f800b040).stop 2021-11-20T05:31:12.131 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:12.130+0000 7fa50e332700 1 -- 172.21.15.58:0/1247106512 shutdown_connections 2021-11-20T05:31:12.132 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:12.130+0000 7fa50e332700 1 --2- 172.21.15.58:0/1247106512 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa4f005c3b0 0x7fa4f005e870 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:12.132 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:12.130+0000 7fa50e332700 1 --2- 172.21.15.58:0/1247106512 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa5080f5410 0x7fa5081094c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:12.132 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:12.130+0000 7fa50e332700 1 --2- 172.21.15.58:0/1247106512 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa508060220 0x7fa50810b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:31:12.132 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:12.130+0000 7fa50e332700 1 -- 172.21.15.58:0/1247106512 >> 172.21.15.58:0/1247106512 conn(0x7fa5080f0b70 msgr2=0x7fa5080f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:31:12.132 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:12.130+0000 7fa50e332700 1 -- 172.21.15.58:0/1247106512 shutdown_connections 2021-11-20T05:31:12.133 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:12.131+0000 7fa50e332700 1 -- 172.21.15.58:0/1247106512 wait complete. 2021-11-20T05:31:12.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:12 smithi058 conmon[33537]: audit 2021-11-20T05:31:11.287208 2021-11-20T05:31:12.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:12 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 186 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:12.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:12 smithi058 conmon[33537]: audit 2021-11-20T05:31:11.288147+0000 mon.smithi058 (mon.0) 187 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:31:12.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:12 smithi058 conmon[33537]: audit 2021-11-20T05:31:12.129636+0000 mon.smithi058 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.58:0/1247106512' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:12.507 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2021-11-20T05:31:12.510 DEBUG:teuthology.orchestra.run.smithi058:> set -ex 2021-11-20T05:31:12.511 DEBUG:teuthology.orchestra.run.smithi058:> sudo dd of=/etc/ceph/ceph.conf 2021-11-20T05:31:12.541 DEBUG:teuthology.orchestra.run.smithi058:> set -ex 2021-11-20T05:31:12.541 DEBUG:teuthology.orchestra.run.smithi058:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-11-20T05:31:12.610 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2021-11-20T05:31:12.610 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.conf 2021-11-20T05:31:12.636 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:12 smithi187 conmon[36358]: audit 2021-11-20T05:31:11.287208+0000 mon.smithi058 (mon.0) 186 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:12.638 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:12 smithi187 conmon[36358]: audit 2021-11-20T05:31:11.288147+0000 mon.smithi058 (mon.0) 187 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:31:12.638 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:12 smithi187 conmon[36358]: audit 2021-11-20T05:31:12.129636+0000 mon.smithi058 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.58:0/1247106512' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:12.640 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2021-11-20T05:31:12.640 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-11-20T05:31:12.707 INFO:tasks.cephadm:Deploying OSDs... 2021-11-20T05:31:12.708 DEBUG:teuthology.orchestra.run.smithi058:> set -ex 2021-11-20T05:31:12.709 DEBUG:teuthology.orchestra.run.smithi058:> dd if=/scratch_devs of=/dev/stdout 2021-11-20T05:31:12.727 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-11-20T05:31:12.727 DEBUG:teuthology.orchestra.run.smithi058:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-11-20T05:31:12.783 INFO:teuthology.orchestra.run.smithi058.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-11-20T05:31:12.784 INFO:teuthology.orchestra.run.smithi058.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T05:31:12.784 INFO:teuthology.orchestra.run.smithi058.stdout:Device: 6h/6d Inode: 119007 Links: 1 2021-11-20T05:31:12.784 INFO:teuthology.orchestra.run.smithi058.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T05:31:12.784 INFO:teuthology.orchestra.run.smithi058.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T05:31:12.785 INFO:teuthology.orchestra.run.smithi058.stdout:Access: 2021-11-20 05:30:40.266051703 +0000 2021-11-20T05:31:12.785 INFO:teuthology.orchestra.run.smithi058.stdout:Modify: 2021-11-20 05:24:21.204382245 +0000 2021-11-20T05:31:12.785 INFO:teuthology.orchestra.run.smithi058.stdout:Change: 2021-11-20 05:24:21.204382245 +0000 2021-11-20T05:31:12.785 INFO:teuthology.orchestra.run.smithi058.stdout: Birth: - 2021-11-20T05:31:12.794 INFO:teuthology.orchestra.run.smithi058.stderr:1+0 records in 2021-11-20T05:31:12.794 INFO:teuthology.orchestra.run.smithi058.stderr:1+0 records out 2021-11-20T05:31:12.794 INFO:teuthology.orchestra.run.smithi058.stderr:512 bytes copied, 0.00010184 s, 5.0 MB/s 2021-11-20T05:31:12.797 DEBUG:teuthology.orchestra.run.smithi058:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-11-20T05:31:12.853 INFO:teuthology.orchestra.run.smithi058.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-11-20T05:31:12.854 INFO:teuthology.orchestra.run.smithi058.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T05:31:12.854 INFO:teuthology.orchestra.run.smithi058.stdout:Device: 6h/6d Inode: 116308 Links: 1 2021-11-20T05:31:12.855 INFO:teuthology.orchestra.run.smithi058.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T05:31:12.855 INFO:teuthology.orchestra.run.smithi058.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T05:31:12.855 INFO:teuthology.orchestra.run.smithi058.stdout:Access: 2021-11-20 05:30:40.266051703 +0000 2021-11-20T05:31:12.856 INFO:teuthology.orchestra.run.smithi058.stdout:Modify: 2021-11-20 05:24:21.562376484 +0000 2021-11-20T05:31:12.856 INFO:teuthology.orchestra.run.smithi058.stdout:Change: 2021-11-20 05:24:21.562376484 +0000 2021-11-20T05:31:12.856 INFO:teuthology.orchestra.run.smithi058.stdout: Birth: - 2021-11-20T05:31:12.864 INFO:teuthology.orchestra.run.smithi058.stderr:1+0 records in 2021-11-20T05:31:12.865 INFO:teuthology.orchestra.run.smithi058.stderr:1+0 records out 2021-11-20T05:31:12.865 INFO:teuthology.orchestra.run.smithi058.stderr:512 bytes copied, 8.0658e-05 s, 6.3 MB/s 2021-11-20T05:31:12.867 DEBUG:teuthology.orchestra.run.smithi058:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-11-20T05:31:12.925 INFO:teuthology.orchestra.run.smithi058.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-11-20T05:31:12.925 INFO:teuthology.orchestra.run.smithi058.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T05:31:12.925 INFO:teuthology.orchestra.run.smithi058.stdout:Device: 6h/6d Inode: 119889 Links: 1 2021-11-20T05:31:12.926 INFO:teuthology.orchestra.run.smithi058.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T05:31:12.926 INFO:teuthology.orchestra.run.smithi058.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T05:31:12.926 INFO:teuthology.orchestra.run.smithi058.stdout:Access: 2021-11-20 05:30:40.266051703 +0000 2021-11-20T05:31:12.926 INFO:teuthology.orchestra.run.smithi058.stdout:Modify: 2021-11-20 05:24:21.904370981 +0000 2021-11-20T05:31:12.926 INFO:teuthology.orchestra.run.smithi058.stdout:Change: 2021-11-20 05:24:21.904370981 +0000 2021-11-20T05:31:12.927 INFO:teuthology.orchestra.run.smithi058.stdout: Birth: - 2021-11-20T05:31:12.936 INFO:teuthology.orchestra.run.smithi058.stderr:1+0 records in 2021-11-20T05:31:12.936 INFO:teuthology.orchestra.run.smithi058.stderr:1+0 records out 2021-11-20T05:31:12.936 INFO:teuthology.orchestra.run.smithi058.stderr:512 bytes copied, 0.000106346 s, 4.8 MB/s 2021-11-20T05:31:12.939 DEBUG:teuthology.orchestra.run.smithi058:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-11-20T05:31:12.995 INFO:teuthology.orchestra.run.smithi058.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-11-20T05:31:12.995 INFO:teuthology.orchestra.run.smithi058.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T05:31:12.995 INFO:teuthology.orchestra.run.smithi058.stdout:Device: 6h/6d Inode: 120860 Links: 1 2021-11-20T05:31:12.996 INFO:teuthology.orchestra.run.smithi058.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T05:31:12.996 INFO:teuthology.orchestra.run.smithi058.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T05:31:12.996 INFO:teuthology.orchestra.run.smithi058.stdout:Access: 2021-11-20 05:30:40.266051703 +0000 2021-11-20T05:31:12.996 INFO:teuthology.orchestra.run.smithi058.stdout:Modify: 2021-11-20 05:24:22.280364931 +0000 2021-11-20T05:31:12.996 INFO:teuthology.orchestra.run.smithi058.stdout:Change: 2021-11-20 05:24:22.280364931 +0000 2021-11-20T05:31:12.997 INFO:teuthology.orchestra.run.smithi058.stdout: Birth: - 2021-11-20T05:31:13.005 INFO:teuthology.orchestra.run.smithi058.stderr:1+0 records in 2021-11-20T05:31:13.005 INFO:teuthology.orchestra.run.smithi058.stderr:1+0 records out 2021-11-20T05:31:13.005 INFO:teuthology.orchestra.run.smithi058.stderr:512 bytes copied, 0.000102226 s, 5.0 MB/s 2021-11-20T05:31:13.008 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2021-11-20T05:31:13.008 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2021-11-20T05:31:13.024 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-11-20T05:31:13.024 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-11-20T05:31:13.080 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-11-20T05:31:13.080 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T05:31:13.081 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 117198 Links: 1 2021-11-20T05:31:13.081 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T05:31:13.081 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T05:31:13.081 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2021-11-20 05:31:08.757914061 +0000 2021-11-20T05:31:13.081 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2021-11-20 05:24:13.067571104 +0000 2021-11-20T05:31:13.082 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2021-11-20 05:24:13.067571104 +0000 2021-11-20T05:31:13.082 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2021-11-20T05:31:13.090 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2021-11-20T05:31:13.091 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2021-11-20T05:31:13.091 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000228783 s, 2.2 MB/s 2021-11-20T05:31:13.093 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-11-20T05:31:13.149 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-11-20T05:31:13.149 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T05:31:13.149 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 119144 Links: 1 2021-11-20T05:31:13.150 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T05:31:13.150 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T05:31:13.150 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2021-11-20 05:31:08.757914061 +0000 2021-11-20T05:31:13.150 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2021-11-20 05:24:13.408565853 +0000 2021-11-20T05:31:13.151 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2021-11-20 05:24:13.408565853 +0000 2021-11-20T05:31:13.151 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2021-11-20T05:31:13.159 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2021-11-20T05:31:13.159 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2021-11-20T05:31:13.159 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000199194 s, 2.6 MB/s 2021-11-20T05:31:13.161 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-11-20T05:31:13.217 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-11-20T05:31:13.217 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T05:31:13.218 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 115479 Links: 1 2021-11-20T05:31:13.218 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T05:31:13.218 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T05:31:13.219 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2021-11-20 05:31:08.757914061 +0000 2021-11-20T05:31:13.219 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2021-11-20 05:24:13.768560309 +0000 2021-11-20T05:31:13.219 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2021-11-20 05:24:13.768560309 +0000 2021-11-20T05:31:13.220 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2021-11-20T05:31:13.227 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2021-11-20T05:31:13.227 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2021-11-20T05:31:13.227 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000198356 s, 2.6 MB/s 2021-11-20T05:31:13.231 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-11-20T05:31:13.287 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-11-20T05:31:13.288 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T05:31:13.288 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 116004 Links: 1 2021-11-20T05:31:13.288 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T05:31:13.289 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T05:31:13.289 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2021-11-20 05:31:08.757914061 +0000 2021-11-20T05:31:13.289 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2021-11-20 05:24:14.094555289 +0000 2021-11-20T05:31:13.290 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2021-11-20 05:24:14.094555289 +0000 2021-11-20T05:31:13.290 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2021-11-20T05:31:13.297 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2021-11-20T05:31:13.298 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2021-11-20T05:31:13.298 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000255165 s, 2.0 MB/s 2021-11-20T05:31:13.300 INFO:tasks.cephadm:Deploying osd.0 on smithi058 with /dev/vg_nvme/lv_4... 2021-11-20T05:31:13.301 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-11-20T05:31:13.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:13 smithi187 conmon[36358]: cluster 2021-11-20T05:31:12.179213+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:31:13.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:13 smithi187 conmon[36358]: 14164) 46 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:13.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:13 smithi058 conmon[33537]: cluster 2021-11-20T05:31:12.179213+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:31:13.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:13 smithi058 conmon[33537]: mgr.14164) 46 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:14.869 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:14 smithi058 conmon[33537]: audit 2021-11-20T05:31:13.630753+0000 mon.smithi058 2021-11-20T05:31:14.869 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:14 smithi058 conmon[33537]: (mon.0) 189 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:14.870 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:14 smithi058 conmon[33537]: cluster 2021-11-20T05:31:14.179593+0000 mgr.smithi058.zagdlk (mgr.14164) 47 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:14.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:14 smithi187 conmon[36358]: audit 2021-11-20T05:31:13.630753+0000 mon.smithi058 (mon.0) 189 : audit [INF] 2021-11-20T05:31:14.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:14 smithi187 conmon[36358]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:14.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:14 smithi187 conmon[36358]: cluster 2021-11-20T05:31:14.179593+0000 mgr.smithi058.zagdlk (mgr.14164) 47 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:15.548 INFO:teuthology.orchestra.run.smithi058.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-11-20T05:31:15.548 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-11-20T05:31:15.548 INFO:teuthology.orchestra.run.smithi058.stderr: stderr: 10+0 records in 2021-11-20T05:31:15.549 INFO:teuthology.orchestra.run.smithi058.stderr:10+0 records out 2021-11-20T05:31:15.549 INFO:teuthology.orchestra.run.smithi058.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00801206 s, 1.3 GB/s 2021-11-20T05:31:15.549 INFO:teuthology.orchestra.run.smithi058.stderr: stderr: 2021-11-20T05:31:15.549 INFO:teuthology.orchestra.run.smithi058.stderr:--> Zapping successful for: 2021-11-20T05:31:15.935 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi058:vg_nvme/lv_4 2021-11-20T05:31:17.425 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:17 smithi058 conmon[33537]: cluster 2021-11-20T05:31:16.179987+0000 mgr.smithi058.zagdlk (mgr.14164) 48 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:17.425 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:17 smithi058 conmon[33537]: audit 2021-11-20T05:31:16.755207+0000 mon.smithi058 (mon.0) 190 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:17.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:17 smithi187 conmon[36358]: cluster 2021-11-20T05:31:16.179987+0000 mgr.smithi058.zagdlk (mgr.14164) 48 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:17.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:17 smithi187 conmon[36358]: audit 2021-11-20T05:31:16.755207+0000 mon.smithi058 (mon.0) 190 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:17.734 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.733+0000 7f0ac909d700 1 -- 172.21.15.58:0/1230158255 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0ac40f66e0 msgr2=0x7f0ac40f6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:17.735 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.733+0000 7f0ac909d700 1 --2- 172.21.15.58:0/1230158255 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0ac40f66e0 0x7f0ac40f6b00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f0ab8004660 tx=0x7f0ab8009b30).stop 2021-11-20T05:31:17.735 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.733+0000 7f0ac909d700 1 -- 172.21.15.58:0/1230158255 shutdown_connections 2021-11-20T05:31:17.736 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.733+0000 7f0ac909d700 1 --2- 172.21.15.58:0/1230158255 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ac40f7980 0x7f0ac40f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:31:17.736 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.733+0000 7f0ac909d700 1 --2- 172.21.15.58:0/1230158255 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0ac40f66e0 0x7f0ac40f6b00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:17.736 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.733+0000 7f0ac909d700 1 -- 172.21.15.58:0/1230158255 >> 172.21.15.58:0/1230158255 conn(0x7f0ac40f1e40 msgr2=0x7f0ac40f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:31:17.737 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.733+0000 7f0ac909d700 1 -- 172.21.15.58:0/1230158255 shutdown_connections 2021-11-20T05:31:17.737 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.733+0000 7f0ac909d700 1 -- 172.21.15.58:0/1230158255 wait complete. 2021-11-20T05:31:17.737 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.734+0000 7f0ac909d700 1 Processor -- start 2021-11-20T05:31:17.737 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.734+0000 7f0ac909d700 1 -- start start 2021-11-20T05:31:17.738 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.734+0000 7f0ac909d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ac40f66e0 0x7f0ac4106f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:31:17.738 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.734+0000 7f0ac909d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0ac40f7980 0x7f0ac4107450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:31:17.738 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.734+0000 7f0ac909d700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac4107a90 con 0x7f0ac40f7980 2021-11-20T05:31:17.738 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.734+0000 7f0ac909d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac4108340 con 0x7f0ac40f66e0 2021-11-20T05:31:17.739 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.734+0000 7f0ac259c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0ac40f7980 0x7f0ac4107450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:31:17.739 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.734+0000 7f0ac259c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0ac40f7980 0x7f0ac4107450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:39970/0 (socket says 172.21.15.58:39970) 2021-11-20T05:31:17.739 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.734+0000 7f0ac259c700 1 -- 172.21.15.58:0/710640125 learned_addr learned my addr 172.21.15.58:0/710640125 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:31:17.739 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.734+0000 7f0ac2d9d700 1 --2- 172.21.15.58:0/710640125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ac40f66e0 0x7f0ac4106f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:31:17.739 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.734+0000 7f0ac259c700 1 -- 172.21.15.58:0/710640125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ac40f66e0 msgr2=0x7f0ac4106f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:17.740 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.734+0000 7f0ac259c700 1 --2- 172.21.15.58:0/710640125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ac40f66e0 0x7f0ac4106f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:17.740 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.734+0000 7f0ac259c700 1 -- 172.21.15.58:0/710640125 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab8005040 con 0x7f0ac40f7980 2021-11-20T05:31:17.740 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.734+0000 7f0ac259c700 1 --2- 172.21.15.58:0/710640125 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0ac40f7980 0x7f0ac4107450 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0ab400b410 tx=0x7f0ab4006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:31:17.740 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.734+0000 7f0aab7fe700 1 -- 172.21.15.58:0/710640125 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ab40125f0 con 0x7f0ac40f7980 2021-11-20T05:31:17.741 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.735+0000 7f0aab7fe700 1 -- 172.21.15.58:0/710640125 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ab4012c00 con 0x7f0ac40f7980 2021-11-20T05:31:17.741 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.735+0000 7f0ac909d700 1 -- 172.21.15.58:0/710640125 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac4108620 con 0x7f0ac40f7980 2021-11-20T05:31:17.741 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.735+0000 7f0aab7fe700 1 -- 172.21.15.58:0/710640125 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ab4002aa0 con 0x7f0ac40f7980 2021-11-20T05:31:17.741 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.735+0000 7f0ac909d700 1 -- 172.21.15.58:0/710640125 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac4108be0 con 0x7f0ac40f7980 2021-11-20T05:31:17.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.736+0000 7f0ac909d700 1 -- 172.21.15.58:0/710640125 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac40f5ad0 con 0x7f0ac40f7980 2021-11-20T05:31:17.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.736+0000 7f0aab7fe700 1 -- 172.21.15.58:0/710640125 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0ab400f030 con 0x7f0ac40f7980 2021-11-20T05:31:17.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.736+0000 7f0aab7fe700 1 --2- 172.21.15.58:0/710640125 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0aac05c2a0 0x7f0aac05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:31:17.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.736+0000 7f0aab7fe700 1 -- 172.21.15.58:0/710640125 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ab4075390 con 0x7f0ac40f7980 2021-11-20T05:31:17.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.736+0000 7f0ac2d9d700 1 --2- 172.21.15.58:0/710640125 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0aac05c2a0 0x7f0aac05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:31:17.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.737+0000 7f0ac2d9d700 1 --2- 172.21.15.58:0/710640125 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0aac05c2a0 0x7f0aac05e760 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f0ab8004e30 tx=0x7f0ab8009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:31:17.745 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.740+0000 7f0aab7fe700 1 -- 172.21.15.58:0/710640125 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ab4049000 con 0x7f0ac40f7980 2021-11-20T05:31:17.926 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:17.927+0000 7f0ac909d700 1 -- 172.21.15.58:0/710640125 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi058:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ac40008d0 con 0x7f0aac05c2a0 2021-11-20T05:31:18.376 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:18 smithi058 conmon[33537]: audit 2021-11-20T05:31:17.312203+0000 mon.smithi058 (mon.0) 191 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:18.376 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:18 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:17.314587+0000 mgr.smithi058.zagdlk (mgr.14164) 49 : cephadm [INF] Reconfiguring mon.smithi058 (unknown last config time)... 2021-11-20T05:31:18.376 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:18 smithi058 conmon[33537]: audit 2021-11-20T05:31:17.314912+0000 mon.smithi058 (mon.0) 192 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T05:31:18.377 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:18 smithi058 conmon[33537]: audit 2021-11-20T05:31:17.315838+0000 mon.smithi058 (mon.0) 193 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-20T05:31:18.377 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:18 smithi058 conmon[33537]: audit 2021-11-20T05:31:17.316519+0000 mon.smithi058 (mon.0) 194 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:18.377 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:18 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:17.317100+0000 mgr.smithi058.zagdlk (mgr.14164) 50 : cephadm [INF] Reconfiguring daemon mon.smithi058 on smithi058 2021-11-20T05:31:18.377 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:18 smithi058 conmon[33537]: audit 2021-11- 2021-11-20T05:31:18.378 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:18 smithi058 conmon[33537]: 20T05:31:17.930063+0000 mon.smithi058 (mon.0) 195 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T05:31:18.378 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:18 smithi058 conmon[33537]: audit 2021-11-20T05:31:17.931662+0000 mon.smithi058 (mon.0) 196 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T05:31:18.378 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:18 smithi058 conmon[33537]: audit 2021-11-20T05:31:17.932261+0000 mon.smithi058 (mon.0) 197 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:18.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: audit 2021-11-20T05:31:17.312203+0000 mon.smithi058 (mon.0) 191 : audit [INF] 2021-11-20T05:31:18.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:18.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:17.314587+0000 mgr.smithi058.zagdlk (mgr.14164) 49 : cephadm [INF] Reconfiguring mon.smithi058 (unknown last config time)... 2021-11-20T05:31:18.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: audit 2021 2021-11-20T05:31:18.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: -11-20T05:31:17. 2021-11-20T05:31:18.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: 314912+0000 mon.smithi058 (mon.0) 192 : 2021-11-20T05:31:18.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T05:31:18.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: audit 2021-11 2021-11-20T05:31:18.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: -20T05:31:17.315838+0000 mon.smithi058 (mon 2021-11-20T05:31:18.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: .0) 193 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-20T05:31:18.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: 2021-11-20T05:31:18.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: audit 2021-11 2021-11-20T05:31:18.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: -20T05:31:17.316519 2021-11-20T05:31:18.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 194 : audit 2021-11-20T05:31:18.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:18.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: cephadm 2021- 2021-11-20T05:31:18.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: 11-20T05:31:17.317100+0000 2021-11-20T05:31:18.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: mgr.smithi058.zagdlk (mgr.14164) 50 : 2021-11-20T05:31:18.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: cephadm [INF] Reconfiguring daemon mon.smithi058 on smithi058 2021-11-20T05:31:18.702 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: audit 2021 2021-11-20T05:31:18.702 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: -11-20T05:31:17.930063 2021-11-20T05:31:18.703 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 195 : 2021-11-20T05:31:18.703 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T05:31:18.703 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: audit 2021 2021-11-20T05:31:18.703 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: -11-20T05:31:17.931662 2021-11-20T05:31:18.704 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 196 2021-11-20T05:31:18.704 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T05:31:18.704 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: audit 2021 2021-11-20T05:31:18.704 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: -11-20T05:31:17.932261 2021-11-20T05:31:18.704 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 197 2021-11-20T05:31:18.705 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:18 smithi187 conmon[36358]: : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:19.567 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:19 smithi058 conmon[33537]: audit 2021-11-20T05:31:17.929201 2021-11-20T05:31:19.567 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:19 smithi058 conmon[33537]: +0000 mgr.smithi058.zagdlk (mgr.14164) 51 : audit [DBG] from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi058:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:31:19.570 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:19 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:17.931431+0000 mgr.smithi058.zagdlk (mgr.14164) 52 : cephadm [INF] Applying service osd.None on host smithi058... 2021-11-20T05:31:19.570 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:19 smithi058 conmon[33537]: cluster 2021-11-20T05:31:18.180238+0000 mgr.smithi058.zagdlk (mgr.14164) 53 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:19.570 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:19 smithi058 conmon[33537]: audit 2021-11-20T05:31:18.655718+0000 mon.smithi058 (mon.0) 198 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:19.571 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:19 smithi058 conmon[33537]: audit 2021-11-20T05:31:18.656515+0000 mon.smithi058 (mon.0) 199 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi058.zagdlk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T05:31:19.571 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:19 smithi058 conmon[33537]: audit 2021-11-20T05:31:18.657216+0000 mon.smithi058 (mon.0) 200 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T05:31:19.571 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:19 smithi058 conmon[33537]: audit 2021-11-20T05:31:18.657800+0000 mon.smithi058 (mon.0) 201 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:19.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:19 smithi187 conmon[36358]: audit 2021-11-20T05:31:17.929201+0000 mgr.smithi058.zagdlk (mgr.14164) 51 : audit [DBG] from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi058:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:31:19.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:19 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:17.931431+0000 mgr.smithi058.zagdlk (mgr.14164) 52 : cephadm [INF] Applying service osd.None on host smithi058... 2021-11-20T05:31:19.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:19 smithi187 conmon[36358]: cluster 2021-11-20T05:31:18.180238+0000 2021-11-20T05:31:19.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:19 smithi187 conmon[36358]: mgr.smithi058.zagdlk (mgr.14164) 53 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:19.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:19 smithi187 conmon[36358]: audit 2021- 2021-11-20T05:31:19.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:19 smithi187 conmon[36358]: 11-20T05:31:18.655718+0000 mon.smithi058 2021-11-20T05:31:19.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:19 smithi187 conmon[36358]: (mon.0) 198 : audit [INF] 2021-11-20T05:31:19.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:19 smithi187 conmon[36358]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:19.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:19 smithi187 conmon[36358]: audit 2021 2021-11-20T05:31:19.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:19 smithi187 conmon[36358]: -11-20T05:31:18.656515 2021-11-20T05:31:19.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:19 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 2021-11-20T05:31:19.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:19 smithi187 conmon[36358]: 199 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi058.zagdlk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T05:31:19.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:19 smithi187 conmon[36358]: audit 2021-11-20T05:31:18.657216+0000 mon.smithi058 (mon.0) 200 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T05:31:19.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:19 smithi187 conmon[36358]: audit 2021-11-20T05:31:18.657800+0000 mon.smithi058 (mon.0) 201 : audit [DBG] 2021-11-20T05:31:19.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:19 smithi187 conmon[36358]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:20.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:20 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:18.656226+0000 mgr.smithi058.zagdlk (mgr.14164) 54 : cephadm [INF] Reconfiguring mgr.smithi058.zagdlk (unknown last config time)... 2021-11-20T05:31:20.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:20 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:18.658332+0000 mgr.smithi058.zagdlk (mgr.14164) 55 : cephadm [INF] Reconfiguring daemon mgr.smithi058.zagdlk on smithi058 2021-11-20T05:31:20.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:20 smithi187 conmon[36358]: audit 2021-11-20T05 2021-11-20T05:31:20.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:20 smithi187 conmon[36358]: :31:20.200859+0000 mon.smithi058 (mon.0) 202 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:20.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:20 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:18.656226+0000 mgr.smithi058.zagdlk (mgr.14164) 54 : cephadm [INF] Reconfiguring mgr.smithi058.zagdlk (unknown last config time)... 2021-11-20T05:31:20.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:20 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:18.658332+0000 mgr.smithi058.zagdlk (mgr.14164) 55 : cephadm [INF] Reconfiguring daemon mgr.smithi058.zagdlk on smithi058 2021-11-20T05:31:20.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:20 smithi058 conmon[33537]: audit 2021-11-20T05:31:20.200859+0000 mon.smithi058 (mon.0) 202 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:21.613 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:21 smithi058 conmon[33537]: cluster 2021-11-20T05:31:20.180500+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:31:21.614 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:21 smithi058 conmon[33537]: mgr.14164) 56 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:21.614 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:21 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:20.201451+0000 mgr.smithi058.zagdlk (mgr.14164) 57 : cephadm [INF] Reconfiguring alertmanager.smithi058 (dependencies changed)... 2021-11-20T05:31:21.614 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:21 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:20.203827+0000 mgr.smithi058.zagdlk (mgr.14164) 58 : cephadm [INF] Reconfiguring daemon alertmanager.smithi058 on smithi058 2021-11-20T05:31:21.614 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:21 smithi058 conmon[33537]: audit 2021-11-20T05:31:21.179781+0000 mon.smithi058 (mon.0) 203 : audit [INF] from='client.? 172.21.15.58:0/1805855485' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2f5103a-e0d2-4bc3-88e1-f28420c98071"}]: dispatch 2021-11-20T05:31:21.615 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:21 smithi058 conmon[33537]: audit 2021-11-20T05:31:21.183136+0000 mon.smithi058 (mon.0) 204 : audit [INF] from='client.? 172.21.15.58:0/1805855485' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2f5103a-e0d2-4bc3-88e1-f28420c98071"}]': finished 2021-11-20T05:31:21.615 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:21 smithi058 conmon[33537]: cluster 2021-11-20T05:31:21.183180+0000 mon.smithi058 (mon.0) 205 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-11-20T05:31:21.615 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:21 smithi058 conmon[33537]: audit 2021-11-20T05:31:21.183267+0000 mon.smithi058 (mon.0) 206 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T05:31:21.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:21 smithi187 conmon[36358]: cluster 2021-11-20T05:31:20.180500+0000 mgr.smithi058.zagdlk (mgr.14164) 56 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:21.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:21 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:20.201451+0000 mgr.smithi058.zagdlk (mgr.14164) 57 : cephadm [INF] Reconfiguring alertmanager.smithi058 (dependencies changed)... 2021-11-20T05:31:21.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:21 smithi187 conmon[36358]: cephadm 2021 2021-11-20T05:31:21.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:21 smithi187 conmon[36358]: -11-20T05:31:20.203827+0000 mgr.smithi058.zagdlk (mgr.14164) 58 : cephadm [INF] Reconfiguring daemon alertmanager.smithi058 on smithi058 2021-11-20T05:31:21.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:21 smithi187 conmon[36358]: audit 2021-11 2021-11-20T05:31:21.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:21 smithi187 conmon[36358]: -20T05:31:21.179781+0000 mon.smithi058 (mon 2021-11-20T05:31:21.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:21 smithi187 conmon[36358]: .0) 203 : audit [INF] from='client.? 172.21.15.58:0/1805855485' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2f5103a-e0d2-4bc3-88e1-f28420c98071"}]: dispatch 2021-11-20T05:31:21.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:21 smithi187 conmon[36358]: audit 2021-11 2021-11-20T05:31:21.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:21 smithi187 conmon[36358]: -20T05:31:21.183136+0000 mon.smithi058 (mon 2021-11-20T05:31:21.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:21 smithi187 conmon[36358]: .0) 204 : audit [INF] from='client.? 172.21.15.58:0/1805855485' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2f5103a-e0d2-4bc3-88e1-f28420c98071"}]': finished 2021-11-20T05:31:21.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:21 smithi187 conmon[36358]: cluster 2021-11 2021-11-20T05:31:21.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:21 smithi187 conmon[36358]: -20T05:31:21.183180+0000 mon.smithi058 ( 2021-11-20T05:31:21.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:21 smithi187 conmon[36358]: mon.0) 205 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-11-20T05:31:21.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:21 smithi187 conmon[36358]: audit 2021 2021-11-20T05:31:21.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:21 smithi187 conmon[36358]: -11-20T05:31:21.183267+0000 2021-11-20T05:31:21.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:21 smithi187 conmon[36358]: mon.smithi058 (mon.0) 206 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T05:31:22.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:22 smithi187 conmon[36358]: audit 2021-11-20T05:31:21.612296+0000 mon.smithi058 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.58:0/2810829795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T05:31:22.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:22 smithi058 conmon[33537]: audit 2021-11-20T05:31:21.612296+0000 mon.smithi058 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.58:0/2810829795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T05:31:23.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:23 smithi187 conmon[36358]: cluster 2021-11-20T05:31:22.180939+0000 mgr.smithi058.zagdlk (mgr 2021-11-20T05:31:23.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:23 smithi187 conmon[36358]: .14164) 59 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:23.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:23 smithi058 conmon[33537]: cluster 2021-11-20T05:31:22.180939+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:31:23.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:23 smithi058 conmon[33537]: mgr.14164) 59 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:25.401 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:25 smithi058 conmon[33537]: cluster 2021-11-20T05:31:24.181271+0000 mgr.smithi058.zagdlk (mgr.14164) 60 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:25.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:25 smithi187 conmon[36358]: cluster 2021-11-20T05:31:24.181271+0000 mgr.smithi058.zagdlk (mgr.14164) 60 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:27.437 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:27 smithi058 conmon[33537]: cluster 2021-11-20T05:31:26.181580+0000 mgr.smithi058.zagdlk 2021-11-20T05:31:27.437 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:27 smithi058 conmon[33537]: (mgr.14164) 61 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:27.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:27 smithi187 conmon[36358]: cluster 2021-11-20T05:31:26.181580+0000 mgr.smithi058.zagdlk (mgr.14164) 61 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:29.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:29 smithi187 conmon[36358]: cluster 2021-11-20T05:31:28.181884+0000 mgr.smithi058.zagdlk (mgr.14164) 62 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:29.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:29 smithi187 conmon[36358]: audit 2021-11-20T05:31:28.648783+0000 mon.smithi058 (mon.0) 208 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:29.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:29 smithi187 conmon[36358]: audit 2021-11-20T05:31:28.649557+0000 mon.smithi058 (mon.0) 209 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi058", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T05:31:29.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:29 smithi187 conmon[36358]: audit 2021-11-20T05:31:28 2021-11-20T05:31:29.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:29 smithi187 conmon[36358]: .650207+0000 mon.smithi058 (mon.0) 210 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:29.456 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:29 smithi058 conmon[33537]: cluster 2021-11-20T05:31:28.181884+0000 2021-11-20T05:31:29.456 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:29 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 62 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:29.457 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:29 smithi058 conmon[33537]: audit 2021-11-20T05:31:28.648783+0000 mon.smithi058 (mon.0) 208 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:29.457 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:29 smithi058 conmon[33537]: audit 2021-11-20T05:31:28.649557+0000 mon.smithi058 (mon.0) 209 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi058", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T05:31:29.457 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:29 smithi058 conmon[33537]: audit 2021-11-20T05:31:28.650207+0000 mon.smithi058 (mon.0) 210 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:30.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:30 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:28.649306+0000 mgr.smithi058.zagdlk (mgr.14164) 63 : cephadm [INF] Reconfiguring crash.smithi058 (monmap changed)... 2021-11-20T05:31:30.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:30 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:28.650675+0000 mgr.smithi058.zagdlk (mgr.14164) 64 : cephadm [INF] Reconfiguring daemon crash.smithi058 on smithi058 2021-11-20T05:31:30.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:30 smithi187 conmon[36358]: audit 2021-11-20T05:31:29.627450 2021-11-20T05:31:30.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:30 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 211 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-11-20T05:31:30.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:30 smithi187 conmon[36358]: audit 2021-11-20T05:31:29.628079+0000 mon.smithi058 ( 2021-11-20T05:31:30.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:30 smithi187 conmon[36358]: mon.0) 212 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:30.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:30 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:28.649306+0000 mgr.smithi058.zagdlk (mgr.14164) 63 : cephadm [INF] Reconfiguring crash.smithi058 (monmap changed)... 2021-11-20T05:31:30.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:30 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:28.650675+0000 mgr.smithi058.zagdlk (mgr.14164) 64 : cephadm [INF] Reconfiguring daemon crash.smithi058 on smithi058 2021-11-20T05:31:30.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:30 smithi058 conmon[33537]: audit 2021-11-20T05:31:29.627450+0000 mon.smithi058 (mon.0) 211 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-11-20T05:31:30.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:30 smithi058 conmon[33537]: audit 2021-11-20T05:31:29.628079+0000 mon.smithi058 (mon.0) 212 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:31.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:31 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:29.628578+0000 mgr.smithi058.zagdlk (mgr.14164) 65 : cephadm [INF] Deploying daemon osd.0 on smithi058 2021-11-20T05:31:31.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:31 smithi187 conmon[36358]: cluster 2021-11-20T05:31:30.182137+0000 mgr.smithi058.zagdlk (mgr.14164) 66 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:31.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:31 smithi187 conmon[36358]: audit 2021-11-20T05:31 2021-11-20T05:31:31.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:31 smithi187 conmon[36358]: :30.557741+0000 mon.smithi058 (mon.0) 213 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:31.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:31 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:29.628578+0000 2021-11-20T05:31:31.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:31 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 65 : cephadm [INF] Deploying daemon osd.0 on smithi058 2021-11-20T05:31:31.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:31 smithi058 conmon[33537]: cluster 2021-11-20T05:31:30.182137+0000 mgr.smithi058.zagdlk (mgr.14164) 66 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:31.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:31 smithi058 conmon[33537]: audit 2021-11-20T05:31:30.557741+0000 mon.smithi058 (mon.0) 213 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:32.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:32 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:30.558388+0000 mgr.smithi058.zagdlk (mgr.14164) 67 : cephadm [INF] Reconfiguring grafana.smithi058 (dependencies changed)... 2021-11-20T05:31:32.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:32 smithi187 conmon[36358]: cephadm 2021-11- 2021-11-20T05:31:32.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:32 smithi187 conmon[36358]: 20T05:31:30.562873+0000 mgr.smithi058.zagdlk (mgr.14164) 68 : cephadm [INF] Reconfiguring daemon grafana.smithi058 on smithi058 2021-11-20T05:31:32.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:32 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:30.558388+0000 mgr.smithi058.zagdlk (mgr.14164) 67 : cephadm [INF] Reconfiguring grafana.smithi058 (dependencies changed)... 2021-11-20T05:31:32.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:32 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:30.562873+0000 mgr.smithi058.zagdlk (mgr.14164) 68 : cephadm [INF] Reconfiguring daemon grafana.smithi058 on smithi058 2021-11-20T05:31:33.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:33 smithi058 conmon[33537]: cluster 2021-11-20T05:31: 2021-11-20T05:31:33.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:33 smithi058 conmon[33537]: 32.182459+0000 mgr.smithi058.zagdlk (mgr.14164) 69 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:33.587 INFO:teuthology.orchestra.run.smithi058.stdout:Created osd(s) 0 on host 'smithi058' 2021-11-20T05:31:33.587 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:33.584+0000 7f0aab7fe700 1 -- 172.21.15.58:0/710640125 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0ac40008d0 con 0x7f0aac05c2a0 2021-11-20T05:31:33.588 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:33.586+0000 7f0ac909d700 1 -- 172.21.15.58:0/710640125 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0aac05c2a0 msgr2=0x7f0aac05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:33.588 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:33.586+0000 7f0ac909d700 1 --2- 172.21.15.58:0/710640125 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0aac05c2a0 0x7f0aac05e760 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f0ab8004e30 tx=0x7f0ab8009b30).stop 2021-11-20T05:31:33.588 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:33.586+0000 7f0ac909d700 1 -- 172.21.15.58:0/710640125 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0ac40f7980 msgr2=0x7f0ac4107450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:33.588 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:33.586+0000 7f0ac909d700 1 --2- 172.21.15.58:0/710640125 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0ac40f7980 0x7f0ac4107450 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0ab400b410 tx=0x7f0ab4006ef0).stop 2021-11-20T05:31:33.589 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:33.586+0000 7f0ac909d700 1 -- 172.21.15.58:0/710640125 shutdown_connections 2021-11-20T05:31:33.589 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:33.586+0000 7f0ac909d700 1 --2- 172.21.15.58:0/710640125 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0aac05c2a0 0x7f0aac05e760 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:33.589 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:33.586+0000 7f0ac909d700 1 --2- 172.21.15.58:0/710640125 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ac40f66e0 0x7f0ac4106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:33.589 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:33.586+0000 7f0ac909d700 1 --2- 172.21.15.58:0/710640125 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0ac40f7980 0x7f0ac4107450 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:33.590 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:33.586+0000 7f0ac909d700 1 -- 172.21.15.58:0/710640125 >> 172.21.15.58:0/710640125 conn(0x7f0ac40f1e40 msgr2=0x7f0ac40fb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:31:33.590 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:33.586+0000 7f0ac909d700 1 -- 172.21.15.58:0/710640125 shutdown_connections 2021-11-20T05:31:33.590 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:33.586+0000 7f0ac909d700 1 -- 172.21.15.58:0/710640125 wait complete. 2021-11-20T05:31:33.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:33 smithi187 conmon[36358]: cluster 2021-11-20T05:31:32.182459+0000 mgr.smithi058.zagdlk (mgr.14164) 69 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:33.990 DEBUG:teuthology.orchestra.run.smithi058:osd.0> sudo journalctl -f -n 0 -u ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.0.service 2021-11-20T05:31:33.993 INFO:tasks.cephadm:Deploying osd.1 on smithi058 with /dev/vg_nvme/lv_3... 2021-11-20T05:31:33.994 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-11-20T05:31:34.022 INFO:journalctl@ceph.osd.0.smithi058.stdout:-- Logs begin at Sat 2021-11-20 05:17:50 UTC. -- 2021-11-20T05:31:34.382 INFO:journalctl@ceph.osd.0.smithi058.stdout:Nov 20 05:31:34 smithi058 conmon[50697]: debug 2021-11-20T05:31:34.045+0000 7f7050a2d080 -1 Falling back to public interface 2021-11-20T05:31:34.710 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:34 smithi058 conmon[33537]: audit 2021-11-20T05:31:33. 2021-11-20T05:31:34.711 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:34 smithi058 conmon[33537]: 580568+0000 mon.smithi058 (mon.0) 214 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:34.711 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:34 smithi058 conmon[33537]: audit 2021-11-20T05:31:33.584239+0000 mon.smithi058 (mon.0) 215 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:34.711 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:34 smithi058 conmon[33537]: cluster 2021-11-20T05:31:34.182759+0000 mgr.smithi058.zagdlk (mgr.14164) 70 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:34.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:34 smithi187 conmon[36358]: audit 2021-11-20T05:31:33.580568+0000 mon.smithi058 (mon.0) 214 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:34.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:34 smithi187 conmon[36358]: audit 2021-11-20T05:31:33.584239+0000 mon.smithi058 (mon.0) 215 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:34.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:34 smithi187 conmon[36358]: cluster 2021-11-20T05:31:34.182759 2021-11-20T05:31:34.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:34 smithi187 conmon[36358]: +0000 mgr.smithi058.zagdlk (mgr.14164) 70 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:36.497 INFO:teuthology.orchestra.run.smithi058.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-11-20T05:31:36.498 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-11-20T05:31:36.498 INFO:teuthology.orchestra.run.smithi058.stderr: stderr: 10+0 records in 2021-11-20T05:31:36.498 INFO:teuthology.orchestra.run.smithi058.stderr:10+0 records out 2021-11-20T05:31:36.498 INFO:teuthology.orchestra.run.smithi058.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00900054 s, 1.2 GB/s 2021-11-20T05:31:36.499 INFO:teuthology.orchestra.run.smithi058.stderr:--> Zapping successful for: 2021-11-20T05:31:36.916 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi058:vg_nvme/lv_3 2021-11-20T05:31:37.556 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:37 smithi058 conmon[33537]: cluster 2021-11-20T05:31:36.182957 2021-11-20T05:31:37.556 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:37 smithi058 conmon[33537]: +0000 mgr.smithi058.zagdlk (mgr.14164) 71 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:37.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:37 smithi187 conmon[36358]: cluster 2021-11-20T05:31:36.182957+0000 mgr.smithi058.zagdlk (mgr.14164) 71 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:37.969 INFO:journalctl@ceph.osd.0.smithi058.stdout:Nov 20 05:31:37 smithi058 conmon[50697]: debug 2021-11-20T05:31:37.570+0000 7f7050a2d080 -1 osd.0 0 log_to_monitors {default=true} 2021-11-20T05:31:38.703 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:38 smithi058 conmon[33537]: audit 2021-11-20T05:31:37.485984+0000 mon.smithi058 (mon.0) 216 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:38.704 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:38 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:37.486777+0000 mgr.smithi058.zagdlk (mgr.14164) 72 : cephadm [INF] Reconfiguring prometheus.smithi058 (dependencies changed)... 2021-11-20T05:31:38.704 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:38 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:37.489228+0000 mgr.smithi058.zagdlk (mgr.14164) 73 : cephadm [INF] Reconfiguring daemon prometheus.smithi058 on smithi058 2021-11-20T05:31:38.704 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:38 smithi058 conmon[33537]: audit 2021-11-20T05:31:37.573773+0000 mon.smithi058 (mon.0) 217 : audit [INF] from='osd.0 [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-11-20T05:31:38.704 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:38 smithi058 conmon[33537]: cluster 2021-11-20T05:31:38.183169+0000 mgr.smithi058.zagdlk (mgr.14164) 74 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:38.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:38 smithi187 conmon[36358]: audit 2021-11-20T05:31:37.485984+0000 mon.smithi058 (mon.0) 216 : audit 2021-11-20T05:31:38.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:38 smithi187 conmon[36358]: [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:38.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:38 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:37.486777+0000 mgr.smithi058.zagdlk (mgr.14164) 72 : cephadm [INF] Reconfiguring prometheus.smithi058 (dependencies changed)... 2021-11-20T05:31:38.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:38 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:37.489228+0000 mgr.smithi058.zagdlk (mgr.14164) 73 : cephadm [INF] Reconfiguring daemon prometheus.smithi058 on smithi058 2021-11-20T05:31:38.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:38 smithi187 conmon[36358]: audit 2021-11-20T05:31:37.573773+0000 mon.smithi058 (mon.0) 217 : audit 2021-11-20T05:31:38.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:38 smithi187 conmon[36358]: [INF] from='osd.0 [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-11-20T05:31:38.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:38 smithi187 conmon[36358]: cluster 2021-11-20 2021-11-20T05:31:38.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:38 smithi187 conmon[36358]: T05:31:38.183169+0000 mgr.smithi058.zagdlk (mgr.14164) 74 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:39.101 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.099+0000 7fc3e5f9e700 1 -- 172.21.15.58:0/3408109311 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3e00f5440 msgr2=0x7fc3e00f5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:39.101 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.099+0000 7fc3e5f9e700 1 --2- 172.21.15.58:0/3408109311 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3e00f5440 0x7fc3e00f5860 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc3d4004660 tx=0x7fc3d4009b30).stop 2021-11-20T05:31:39.102 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.099+0000 7fc3e5f9e700 1 -- 172.21.15.58:0/3408109311 shutdown_connections 2021-11-20T05:31:39.102 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.099+0000 7fc3e5f9e700 1 --2- 172.21.15.58:0/3408109311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3e00f66e0 0x7fc3e00f6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:39.102 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.099+0000 7fc3e5f9e700 1 --2- 172.21.15.58:0/3408109311 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3e00f5440 0x7fc3e00f5860 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:39.102 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.099+0000 7fc3e5f9e700 1 -- 172.21.15.58:0/3408109311 >> 172.21.15.58:0/3408109311 conn(0x7fc3e00f0ba0 msgr2=0x7fc3e00f2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:31:39.103 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.099+0000 7fc3e5f9e700 1 -- 172.21.15.58:0/3408109311 shutdown_connections 2021-11-20T05:31:39.103 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.099+0000 7fc3e5f9e700 1 -- 172.21.15.58:0/3408109311 wait complete. 2021-11-20T05:31:39.103 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.100+0000 7fc3e5f9e700 1 Processor -- start 2021-11-20T05:31:39.103 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.100+0000 7fc3e5f9e700 1 -- start start 2021-11-20T05:31:39.104 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.100+0000 7fc3e5f9e700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3e00f5440 0x7fc3e0109ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:31:39.104 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.100+0000 7fc3e5f9e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3e00f66e0 0x7fc3e010bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:31:39.104 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.100+0000 7fc3e5f9e700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e00603c0 con 0x7fc3e00f5440 2021-11-20T05:31:39.104 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.100+0000 7fc3e5f9e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e0060500 con 0x7fc3e00f66e0 2021-11-20T05:31:39.105 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.100+0000 7fc3e4f9c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3e00f5440 0x7fc3e0109ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:31:39.105 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.100+0000 7fc3e4f9c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3e00f5440 0x7fc3e0109ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40014/0 (socket says 172.21.15.58:40014) 2021-11-20T05:31:39.105 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.100+0000 7fc3e4f9c700 1 -- 172.21.15.58:0/3179168663 learned_addr learned my addr 172.21.15.58:0/3179168663 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:31:39.105 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.101+0000 7fc3dffff700 1 --2- 172.21.15.58:0/3179168663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3e00f66e0 0x7fc3e010bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:31:39.106 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.101+0000 7fc3e4f9c700 1 -- 172.21.15.58:0/3179168663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3e00f66e0 msgr2=0x7fc3e010bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:39.106 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.101+0000 7fc3e4f9c700 1 --2- 172.21.15.58:0/3179168663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3e00f66e0 0x7fc3e010bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:39.106 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.101+0000 7fc3e4f9c700 1 -- 172.21.15.58:0/3179168663 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d4005040 con 0x7fc3e00f5440 2021-11-20T05:31:39.107 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.101+0000 7fc3dffff700 1 --2- 172.21.15.58:0/3179168663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3e00f66e0 0x7fc3e010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:31:39.107 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.101+0000 7fc3e4f9c700 1 --2- 172.21.15.58:0/3179168663 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3e00f5440 0x7fc3e0109ae0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fc3d4007a60 tx=0x7fc3d4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:31:39.107 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.101+0000 7fc3dd7fa700 1 -- 172.21.15.58:0/3179168663 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3d4018070 con 0x7fc3e00f5440 2021-11-20T05:31:39.107 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.101+0000 7fc3dd7fa700 1 -- 172.21.15.58:0/3179168663 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3d4013410 con 0x7fc3e00f5440 2021-11-20T05:31:39.108 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.101+0000 7fc3e5f9e700 1 -- 172.21.15.58:0/3179168663 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e010c530 con 0x7fc3e00f5440 2021-11-20T05:31:39.108 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.101+0000 7fc3dd7fa700 1 -- 172.21.15.58:0/3179168663 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3d401c620 con 0x7fc3e00f5440 2021-11-20T05:31:39.108 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.101+0000 7fc3e5f9e700 1 -- 172.21.15.58:0/3179168663 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e010ca60 con 0x7fc3e00f5440 2021-11-20T05:31:39.109 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.102+0000 7fc3e5f9e700 1 -- 172.21.15.58:0/3179168663 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e0059100 con 0x7fc3e00f5440 2021-11-20T05:31:39.110 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.102+0000 7fc3dd7fa700 1 -- 172.21.15.58:0/3179168663 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc3d401d420 con 0x7fc3e00f5440 2021-11-20T05:31:39.110 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.103+0000 7fc3dd7fa700 1 --2- 172.21.15.58:0/3179168663 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc3d005c2f0 0x7fc3d005e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:31:39.110 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.103+0000 7fc3dd7fa700 1 -- 172.21.15.58:0/3179168663 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1476+0+0 (secure 0 0 0) 0x7fc3d4076e30 con 0x7fc3e00f5440 2021-11-20T05:31:39.110 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.103+0000 7fc3dffff700 1 --2- 172.21.15.58:0/3179168663 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc3d005c2f0 0x7fc3d005e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:31:39.111 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.103+0000 7fc3dffff700 1 --2- 172.21.15.58:0/3179168663 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc3d005c2f0 0x7fc3d005e7b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fc3cc009740 tx=0x7fc3cc009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:31:39.111 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.107+0000 7fc3dd7fa700 1 -- 172.21.15.58:0/3179168663 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc3d404aa30 con 0x7fc3e00f5440 2021-11-20T05:31:39.300 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:39.301+0000 7fc3e5f9e700 1 -- 172.21.15.58:0/3179168663 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi058:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3e00fdd50 con 0x7fc3d005c2f0 2021-11-20T05:31:39.695 INFO:journalctl@ceph.osd.0.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[50697]: debug 2021-11-20T05:31:39.493+0000 7f7046f9e700 -1 osd.0 0 waiting for initial osdmap 2021-11-20T05:31:39.695 INFO:journalctl@ceph.osd.0.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[50697]: debug 2021-11-20T05:31:39.497+0000 7f7040714700 -1 osd.0 8 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2021-11-20T05:31:39.696 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: audit 2021-11-20T05:31:38.489791 2021-11-20T05:31:39.696 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 218 : audit [INF] from='osd.0 [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-11-20T05:31:39.696 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: cluster 2021-11-20T05:31:38.489841+0000 mon.smithi058 (mon. 2021-11-20T05:31:39.696 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: 0) 219 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-11-20T05:31:39.696 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: audit 2021-11-20T05:31:38.489969+0000 mon.smithi058 2021-11-20T05:31:39.697 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: (mon.0) 220 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T05:31:39.697 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: audit 2021-11-20T05:31:39.697 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: 2021-11-20T05:31:38.490354+0000 mon.smithi058 (mon.0) 2021-11-20T05:31:39.697 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: 221 : audit [INF] from='osd.0 [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi058", "root=default"]}]: dispatch 2021-11-20T05:31:39.698 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: audit 2021-11-20T05:31: 2021-11-20T05:31:39.698 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: 39.302965+0000 mgr.smithi058.zagdlk (mgr.14164) 75 : audit [DBG] 2021-11-20T05:31:39.698 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi058:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:31:39.698 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: audit 2021-11-20T05:31:39.699 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: 2021-11-20T05:31:39.303723+0000 mon.smithi058 (mon.0) 222 : 2021-11-20T05:31:39.699 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T05:31:39.699 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:39.699 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: 2021-11-20T05:31:39.304930+0000 mgr.smithi058.zagdlk (mgr.14164) 76 2021-11-20T05:31:39.699 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: : cephadm [INF] Applying service osd.None on host smithi058... 2021-11-20T05:31:39.700 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: audit 2021-11-20T05:31:39.700 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: 2021-11-20T05:31:39.305125+0000 2021-11-20T05:31:39.700 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: mon.smithi058 (mon.0) 223 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T05:31:39.700 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: audit 2021-11-20T05: 2021-11-20T05:31:39.700 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: 31:39.305728+0000 mon.smithi058 (mon.0) 224 : audit [DBG] 2021-11-20T05:31:39.701 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:39 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:39.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:39 smithi187 conmon[36358]: audit 2021-11-20T05:31:38.489791+0000 mon.smithi058 (mon.0) 218 : audit [INF] from='osd.0 [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-11-20T05:31:39.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:39 smithi187 conmon[36358]: cluster 2021-11-20T05:31:38.489841+0000 mon.smithi058 (mon.0) 219 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-11-20T05:31:39.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:39 smithi187 conmon[36358]: audit 2021-11-20T05:31:38. 2021-11-20T05:31:39.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:39 smithi187 conmon[36358]: 489969+0000 mon.smithi058 (mon.0) 220 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T05:31:39.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:39 smithi187 conmon[36358]: audit 2021-11-20T05:31:38. 2021-11-20T05:31:39.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:39 smithi187 conmon[36358]: 490354+0000 mon.smithi058 (mon.0) 221 : audit [INF] from='osd.0 [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi058", "root=default"]}]: dispatch 2021-11-20T05:31:39.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:39 smithi187 conmon[36358]: audit 2021-11-20T05:31:39.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:39 smithi187 conmon[36358]: 2021-11-20T05:31:39.302965+0000 mgr.smithi058.zagdlk (mgr.14164) 75 : audit [DBG] from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi058:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:31:39.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:39 smithi187 conmon[36358]: audit 2021-11-20T05:31:39.303723+0000 mon.smithi058 2021-11-20T05:31:39.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:39 smithi187 conmon[36358]: (mon.0) 222 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T05:31:39.950 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:39 smithi187 conmon[36358]: cephadm 2021 2021-11-20T05:31:39.950 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:39 smithi187 conmon[36358]: -11-20T05:31:39.304930+0000 mgr.smithi058.zagdlk (mgr.14164) 76 : cephadm [INF] Applying service osd.None on host smithi058... 2021-11-20T05:31:39.950 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:39 smithi187 conmon[36358]: audit 2021-11-20T05:31:39.305125+0000 2021-11-20T05:31:39.950 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:39 smithi187 conmon[36358]: mon.smithi058 (mon.0) 223 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T05:31:39.951 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:39 smithi187 conmon[36358]: audit 2021-11-20T05:31:39.305728+0000 mon.smithi058 (mon.0 2021-11-20T05:31:39.951 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:39 smithi187 conmon[36358]: ) 224 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:40.614 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: audit 2021-11-20T05:31:39.490764 2021-11-20T05:31:40.614 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 225 : audit [INF] from='osd.0 [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi058", "root=default"]}]': finished 2021-11-20T05:31:40.615 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:31:40.615 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: -11-20T05:31:39.490829+0000 2021-11-20T05:31:40.615 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: mon.smithi058 (mon.0) 226 : cluster [DBG] 2021-11-20T05:31:40.615 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: osdmap e8: 1 total, 0 up, 1 in 2021-11-20T05:31:40.616 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:31:40.616 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: 11-20T05:31: 2021-11-20T05:31:40.616 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: 39.491158+0000 mon.smithi058 (mon.0 2021-11-20T05:31:40.616 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: ) 227 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T05:31:40.617 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:31:40.617 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: 11-20T05:31: 2021-11-20T05:31:40.617 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: 39.494669+0000 mon.smithi058 (mon.0) 228 2021-11-20T05:31:40.617 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T05:31:40.618 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: cluster 2021-11-20T05:31:40.618 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: 2021-11-20T05:31 2021-11-20T05:31:40.618 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: :40.183423+0000 mgr.smithi058.zagdlk (mgr 2021-11-20T05:31:40.618 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: .14164) 77 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:40.619 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: audit 2021-11-20T05:31:40.494093+0000 mon.smithi058 ( 2021-11-20T05:31:40.619 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:40 smithi058 conmon[33537]: mon.0) 229 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T05:31:40.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:40 smithi187 conmon[36358]: audit 2021-11-20T05:31:39.490764+0000 mon.smithi058 (mon.0) 225 : audit [INF] from='osd.0 [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi058", "root=default"]}]': finished 2021-11-20T05:31:40.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:40 smithi187 conmon[36358]: cluster 2021-11-20T05:31:39.490829+0000 mon.smithi058 (mon.0) 226 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2021-11-20T05:31:40.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:40 smithi187 conmon[36358]: audit 2021-11-20T05:31:39.491158+0000 mon.smithi058 (mon.0) 227 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T05:31:40.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:40 smithi187 conmon[36358]: audit 2021-11-20T05:31:39.494669+0000 mon.smithi058 (mon.0) 228 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T05:31:40.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:40 smithi187 conmon[36358]: cluster 2021-11-20T05:31:40.183423+0000 mgr.smithi058.zagdlk (mgr.14164) 77 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:31:40.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:40 smithi187 conmon[36358]: audit 2021-11-20T05:31:40.494093+0000 mon.smithi058 (mon.0) 229 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T05:31:41.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:41 smithi058 conmon[33537]: cluster 2021-11-20T05:31:38.521991+0000 osd.0 (osd.0 2021-11-20T05:31:41.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:41 smithi058 conmon[33537]: ) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T05:31:41.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:41 smithi058 conmon[33537]: cluster 2021-11-20T05:31:38.522092+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T05:31:41.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:41 smithi058 conmon[33537]: cluster 2021-11-20T05:31:40.499627+0000 mon.smithi058 (mon.0 2021-11-20T05:31:41.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:41 smithi058 conmon[33537]: ) 230 : cluster [INF] osd.0 [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844] boot 2021-11-20T05:31:41.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:41 smithi058 conmon[33537]: cluster 2021-11-20T05:31:41.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:41 smithi058 conmon[33537]: 2021-11-20T05:31:40.499666+0000 2021-11-20T05:31:41.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:41 smithi058 conmon[33537]: mon.smithi058 (mon.0) 231 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-11-20T05:31:41.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:41 smithi058 conmon[33537]: audit 2021-11- 2021-11-20T05:31:41.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:41 smithi058 conmon[33537]: 20T05:31:40.499975+0000 mon.smithi058 (mon. 2021-11-20T05:31:41.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:41 smithi058 conmon[33537]: 0) 232 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T05:31:41.932 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:41 smithi187 conmon[36358]: cluster 2021-11-20T05:31:38.521991+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T05:31:41.933 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:41 smithi187 conmon[36358]: cluster 2021-11-20T05:31:38.522092+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T05:31:41.934 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:41 smithi187 conmon[36358]: cluster 2021-11- 2021-11-20T05:31:41.935 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:41 smithi187 conmon[36358]: 20T05:31:40.499627+0000 mon.smithi058 (mon.0) 230 : cluster [INF] osd.0 [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844] boot 2021-11-20T05:31:41.935 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:41 smithi187 conmon[36358]: cluster 2021-11 2021-11-20T05:31:41.935 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:41 smithi187 conmon[36358]: -20T05:31:40.499666+0000 mon.smithi058 (mon.0) 231 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-11-20T05:31:41.936 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:41 smithi187 conmon[36358]: audit 2021-11-20T05:31:40.499975+0000 mon.smithi058 (mon.0) 232 : audit 2021-11-20T05:31:41.936 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:41 smithi187 conmon[36358]: [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T05:31:42.894 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:42 smithi058 conmon[33537]: cluster 2021-11-20T05:31:41.507498+0000 mon.smithi058 (mon.0) 233 : cluster [DBG] osdmap e10: 1 total, 1 up, 1 in 2021-11-20T05:31:42.895 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:42 smithi058 conmon[33537]: audit 2021-11-20T05:31:41.512637+0000 mon.smithi058 (mon.0) 234 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:42.895 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:42 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:41.513202+0000 mgr.smithi058.zagdlk (mgr.14164) 78 : cephadm [INF] Reconfiguring crash.smithi187 (monmap changed)... 2021-11-20T05:31:42.895 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:42 smithi058 conmon[33537]: audit 2021-11-20T05:31:41.513445+0000 mon.smithi058 (mon.0) 235 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi187", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T05:31:42.896 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:42 smithi058 conmon[33537]: audit 2021-11-20T05:31:41.514115+0000 mon.smithi058 (mon.0) 236 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:42.896 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:42 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:41.514606+0000 mgr.smithi058.zagdlk (mgr.14164) 79 : cephadm [INF] Reconfiguring daemon crash.smithi187 on smithi187 2021-11-20T05:31:42.896 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:42 smithi058 conmon[33537]: cluster 2021-11-20T05 2021-11-20T05:31:42.897 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:42 smithi058 conmon[33537]: :31:42.183673+0000 mgr.smithi058.zagdlk (mgr.14164) 80 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:31:42.897 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:42 smithi058 conmon[33537]: audit 2021-11-20T05:31:42.457446+0000 mon.smithi058 (mon.0) 237 : audit [INF] from='client.? 172.21.15.58:0/3591928005' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0243e40-c76e-4c68-8ac0-8f555035e948"}]: dispatch 2021-11-20T05:31:42.897 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:42 smithi058 conmon[33537]: audit 2021-11-20T05:31:42.460960+0000 mon.smithi058 (mon.0) 238 : audit [INF] from='client.? 172.21.15.58:0/3591928005' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0243e40-c76e-4c68-8ac0-8f555035e948"}]': finished 2021-11-20T05:31:42.898 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:42 smithi058 conmon[33537]: cluster 2021-11-20T05:31:42.461013+0000 mon.smithi058 (mon.0) 239 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-11-20T05:31:42.898 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:42 smithi058 conmon[33537]: audit 2021-11-20T05:31:42.461105+0000 mon.smithi058 (mon.0) 240 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T05:31:42.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:42 smithi187 conmon[36358]: cluster 2021-11-20T05:31:41.507498+0000 mon.smithi058 (mon.0) 233 : cluster [DBG] osdmap e10: 1 total, 1 up, 1 in 2021-11-20T05:31:42.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:42 smithi187 conmon[36358]: audit 2021-11-20T05:31:41.512637+0000 mon.smithi058 (mon.0) 234 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:42.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:42 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:41.513202+0000 mgr.smithi058.zagdlk (mgr.14164) 78 : cephadm [INF] Reconfiguring crash.smithi187 (monmap changed)... 2021-11-20T05:31:42.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:42 smithi187 conmon[36358]: audit 2021-11-20T05:31:41.513445+0000 mon.smithi058 (mon.0) 235 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi187", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T05:31:42.950 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:42 smithi187 conmon[36358]: audit 2021-11-20T05:31:41.514115+0000 mon.smithi058 (mon.0) 236 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:42.950 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:42 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:41.514606+0000 mgr.smithi058.zagdlk (mgr.14164) 79 : cephadm [INF] Reconfiguring daemon crash.smithi187 on smithi187 2021-11-20T05:31:42.951 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:42 smithi187 conmon[36358]: cluster 2021-11-20T05:31:42.183673+0000 mgr.smithi058.zagdlk (mgr.14164) 80 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:31:42.951 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:42 smithi187 conmon[36358]: audit 2021-11-20T05:31:42.457446+0000 mon.smithi058 (mon.0) 237 : audit [INF] from='client.? 172.21.15.58:0/3591928005' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0243e40-c76e-4c68-8ac0-8f555035e948"}]: dispatch 2021-11-20T05:31:42.951 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:42 smithi187 conmon[36358]: audit 2021-11 2021-11-20T05:31:42.952 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:42 smithi187 conmon[36358]: -20T05:31:42.460960+0000 mon.smithi058 (mon.0) 238 : audit [INF] from='client.? 172.21.15.58:0/3591928005' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0243e40-c76e-4c68-8ac0-8f555035e948"}]': finished 2021-11-20T05:31:42.952 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:42 smithi187 conmon[36358]: cluster 2021-11-20T05:31:42.461013+0000 mon.smithi058 (mon.0) 239 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-11-20T05:31:42.953 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:42 smithi187 conmon[36358]: audit 2021-11-20T05:31:42.461105+0000 mon.smithi058 (mon.0) 240 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T05:31:43.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:43 smithi187 conmon[36358]: audit 2021-11-20T05:31:42.692841+0000 mon.smithi058 (mon.0) 241 : 2021-11-20T05:31:43.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:43 smithi187 conmon[36358]: audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:43.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:43 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:42.693368+0000 mgr.smithi058.zagdlk (mgr.14164) 81 : cephadm [INF] Reconfiguring mgr.smithi187.omkctn (monmap changed)... 2021-11-20T05:31:43.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:43 smithi187 conmon[36358]: audit 2021-11-20T05:31:42.693605+0000 mon.smithi058 (mon.0) 242 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi187.omkctn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T05:31:43.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:43 smithi187 conmon[36358]: audit 2021-11-20T05:31:42.694377+0000 mon.smithi058 (mon.0) 243 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T05:31:43.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:43 smithi187 conmon[36358]: audit 2021-11-20T05:31:42.695007+0000 mon.smithi058 (mon.0) 244 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:43.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:43 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:42.695534+0000 mgr.smithi058.zagdlk (mgr.14164) 82 : cephadm [INF] Reconfiguring daemon mgr.smithi187.omkctn on smithi187 2021-11-20T05:31:43.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:43 smithi187 conmon[36358]: audit 2021-11-20T05:31:42.894591+0000 mon.smithi058 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.58:0/2355139079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T05:31:43.969 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:43 smithi058 conmon[33537]: audit 2021-11-20T05:31:42.692841+0000 mon.smithi058 (mon.0) 241 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:43.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:43 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:42.693368+0000 mgr.smithi058.zagdlk (mgr.14164) 81 : cephadm [INF] Reconfiguring mgr.smithi187.omkctn (monmap changed)... 2021-11-20T05:31:43.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:43 smithi058 conmon[33537]: audit 2021-11-20T05:31:42.693605+0000 mon.smithi058 (mon.0) 242 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi187.omkctn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T05:31:43.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:43 smithi058 conmon[33537]: audit 2021-11-20T05:31:42.694377+0000 mon.smithi058 (mon.0) 243 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T05:31:43.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:43 smithi058 conmon[33537]: audit 2021-11-20T05:31:42.695007+0000 mon.smithi058 (mon.0) 244 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:43.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:43 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:42.695534+0000 mgr.smithi058.zagdlk (mgr.14164) 82 : cephadm [INF] Reconfiguring daemon mgr.smithi187.omkctn on smithi187 2021-11-20T05:31:43.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:43 smithi058 conmon[33537]: audit 2021-11-20T05:31 2021-11-20T05:31:43.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:43 smithi058 conmon[33537]: :42.894591+0000 mon.smithi058 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.58:0/2355139079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T05:31:45.219 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: audit 2021-11-20T05:31:43.965154+0000 mon.smithi058 (mon.0) 246 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:45.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:43.965733+0000 mgr.smithi058.zagdlk (mgr.14164) 83 : cephadm [INF] Reconfiguring mon.smithi187 (monmap changed)... 2021-11-20T05:31:45.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: audit 2021-11-20T05:31: 2021-11-20T05:31:45.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: 43.966010+0000 mon.smithi058 (mon.0) 247 : audit [INF] 2021-11-20T05:31:45.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T05:31:45.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:31:45.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: 11-20T05:31:43.966753 2021-11-20T05:31:45.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 248 2021-11-20T05:31:45.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-20T05:31:45.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: audit 2021 2021-11-20T05:31:45.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: -11-20T05:31:43 2021-11-20T05:31:45.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: .967484+0000 mon.smithi058 (mon. 2021-11-20T05:31:45.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: 0) 249 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:45.223 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: 2021-11-20T05:31:45.223 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: cephadm 2021 2021-11-20T05:31:45.223 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: -11-20T05:31:43.968104+0000 2021-11-20T05:31:45.223 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 84 2021-11-20T05:31:45.224 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: : cephadm [INF] Reconfiguring daemon mon.smithi187 on smithi187 2021-11-20T05:31:45.224 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: cluster 2021-11-20T05:31:45.224 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: 2021-11-20T05:31 2021-11-20T05:31:45.224 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: :44.183984+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:31:45.225 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: mgr.14164) 85 : cluster 2021-11-20T05:31:45.225 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:44 smithi058 conmon[33537]: [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:31:45.402 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:44 smithi187 conmon[36358]: audit 2021-11-20T05:31:43.965154+0000 mon.smithi058 (mon.0) 246 : audit 2021-11-20T05:31:45.403 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:44 smithi187 conmon[36358]: [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:45.403 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:44 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:43.965733+0000 mgr.smithi058.zagdlk (mgr.14164) 83 : cephadm [INF] Reconfiguring mon.smithi187 (monmap changed)... 2021-11-20T05:31:45.403 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:44 smithi187 conmon[36358]: audit 2021-11-20T05:31:43.966010+0000 mon.smithi058 (mon.0) 247 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T05:31:45.403 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:44 smithi187 conmon[36358]: audit 2021-11-20T05:31:43.966753+0000 mon.smithi058 (mon.0) 248 : audit 2021-11-20T05:31:45.404 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:44 smithi187 conmon[36358]: [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-20T05:31:45.404 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:44 smithi187 conmon[36358]: audit 2021-11-20T05:31:43.967484 2021-11-20T05:31:45.404 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:44 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 249 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:45.404 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:44 smithi187 conmon[36358]: cephadm 2021 2021-11-20T05:31:45.405 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:44 smithi187 conmon[36358]: -11-20T05:31:43 2021-11-20T05:31:45.405 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:44 smithi187 conmon[36358]: .968104+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:31:45.405 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:44 smithi187 conmon[36358]: 84 : cephadm [INF] Reconfiguring daemon mon.smithi187 on smithi187 2021-11-20T05:31:45.405 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:44 smithi187 conmon[36358]: cluster 2021-11-20T05:31:44.183984+0000 mgr.smithi058.zagdlk (mgr 2021-11-20T05:31:45.406 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:44 smithi187 conmon[36358]: .14164) 85 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:31:46.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:46 smithi187 conmon[36358]: audit 2021-11-20T05:31:45.441728+0000 mon.smithi058 (mon.0) 250 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:46.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:46 smithi187 conmon[36358]: audit 2021-11-20T05:31:45.444738+0000 mon.smithi058 (mon.0) 251 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-20T05:31:46.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:46 smithi187 conmon[36358]: audit 2021-11-20T05 2021-11-20T05:31:46.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:46 smithi187 conmon[36358]: :31:45.445207+0000 mgr.smithi058.zagdlk (mgr.14164) 86 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-20T05:31:46.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:46 smithi187 conmon[36358]: cluster 2021-11-20T05:31:46.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:46 smithi187 conmon[36358]: 2021-11-20T05:31:46 2021-11-20T05:31:46.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:46 smithi187 conmon[36358]: .184217+0000 mgr.smithi058.zagdlk (mgr.14164) 87 : 2021-11-20T05:31:46.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:46 smithi187 conmon[36358]: cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:31:46.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:46 smithi058 conmon[33537]: audit 2021-11-20T05:31:45.441728+0000 mon.smithi058 (mon.0) 250 : 2021-11-20T05:31:46.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:46 smithi058 conmon[33537]: audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:46.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:46 smithi058 conmon[33537]: audit 2021-11-20T05:31:45.444738+0000 mon.smithi058 (mon.0) 251 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-20T05:31:46.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:46 smithi058 conmon[33537]: audit 2021-11-20T05:31:45.445207+0000 mgr.smithi058.zagdlk (mgr.14164) 86 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-20T05:31:46.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:46 smithi058 conmon[33537]: cluster 2021-11-20T05:31:46.184217+0000 mgr.smithi058.zagdlk (mgr.14164) 87 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:31:49.457 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:49 smithi058 conmon[33537]: cluster 2021-11-20T05:31:48.184514+0000 mgr.smithi058.zagdlk (mgr.14164) 88 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:31:49.458 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:49 smithi058 conmon[33537]: audit 2021-11-20T05:31:48.609269+0000 mon.smithi058 (mon.0) 252 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-11-20T05:31:49.458 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:49 smithi058 conmon[33537]: audit 2021-11-20T05:31:48.609965+0000 mon.smithi058 (mon.0) 253 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:49.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:49 smithi187 conmon[36358]: cluster 2021-11-20T05:31:48.184514+0000 mgr.smithi058.zagdlk (mgr.14164) 88 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:31:49.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:49 smithi187 conmon[36358]: audit 2021-11-20T05:31:48.609269+0000 mon.smithi058 (mon.0) 252 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-11-20T05:31:49.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:49 smithi187 conmon[36358]: audit 2021-11-20T05:31:48.609965+0000 mon.smithi058 (mon.0) 253 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:50.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:50 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:48 2021-11-20T05:31:50.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:50 smithi058 conmon[33537]: .610529+0000 mgr.smithi058.zagdlk (mgr.14164) 89 : cephadm [INF] Deploying daemon osd.1 on smithi058 2021-11-20T05:31:50.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:50 smithi187 conmon[36358]: cephadm 2021-11-20T05:31:48.610529+0000 mgr.smithi058.zagdlk (mgr 2021-11-20T05:31:50.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:50 smithi187 conmon[36358]: .14164) 89 : cephadm [INF] Deploying daemon osd.1 on smithi058 2021-11-20T05:31:51.533 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:51 smithi058 conmon[33537]: cluster 2021-11-20T05:31:50.184759 2021-11-20T05:31:51.534 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:51 smithi058 conmon[33537]: +0000 mgr.smithi058.zagdlk (mgr.14164) 90 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:31:51.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:51 smithi187 conmon[36358]: cluster 2021-11-20T05:31:50.184759+0000 mgr.smithi058.zagdlk (mgr.14164) 90 : cluster [DBG] 2021-11-20T05:31:51.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:51 smithi187 conmon[36358]: pgmap v55: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:31:51.743 INFO:teuthology.orchestra.run.smithi058.stdout:Created osd(s) 1 on host 'smithi058' 2021-11-20T05:31:51.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:51.741+0000 7fc3dd7fa700 1 -- 172.21.15.58:0/3179168663 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc3e00fdd50 con 0x7fc3d005c2f0 2021-11-20T05:31:51.745 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:51.743+0000 7fc3e5f9e700 1 -- 172.21.15.58:0/3179168663 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc3d005c2f0 msgr2=0x7fc3d005e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:51.745 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:51.743+0000 7fc3e5f9e700 1 --2- 172.21.15.58:0/3179168663 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc3d005c2f0 0x7fc3d005e7b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fc3cc009740 tx=0x7fc3cc009be0).stop 2021-11-20T05:31:51.745 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:51.743+0000 7fc3e5f9e700 1 -- 172.21.15.58:0/3179168663 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3e00f5440 msgr2=0x7fc3e0109ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:51.745 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:51.743+0000 7fc3e5f9e700 1 --2- 172.21.15.58:0/3179168663 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3e00f5440 0x7fc3e0109ae0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fc3d4007a60 tx=0x7fc3d4009b30).stop 2021-11-20T05:31:51.746 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:51.743+0000 7fc3e5f9e700 1 -- 172.21.15.58:0/3179168663 shutdown_connections 2021-11-20T05:31:51.746 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:51.743+0000 7fc3e5f9e700 1 --2- 172.21.15.58:0/3179168663 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc3d005c2f0 0x7fc3d005e7b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:51.746 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:51.743+0000 7fc3e5f9e700 1 --2- 172.21.15.58:0/3179168663 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3e00f5440 0x7fc3e0109ae0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:51.746 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:51.743+0000 7fc3e5f9e700 1 --2- 172.21.15.58:0/3179168663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3e00f66e0 0x7fc3e010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:51.747 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:51.743+0000 7fc3e5f9e700 1 -- 172.21.15.58:0/3179168663 >> 172.21.15.58:0/3179168663 conn(0x7fc3e00f0ba0 msgr2=0x7fc3e00fa130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:31:51.747 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:51.744+0000 7fc3e5f9e700 1 -- 172.21.15.58:0/3179168663 shutdown_connections 2021-11-20T05:31:51.747 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:51.744+0000 7fc3e5f9e700 1 -- 172.21.15.58:0/3179168663 wait complete. 2021-11-20T05:31:52.118 DEBUG:teuthology.orchestra.run.smithi058:osd.1> sudo journalctl -f -n 0 -u ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.1.service 2021-11-20T05:31:52.121 INFO:tasks.cephadm:Deploying osd.2 on smithi058 with /dev/vg_nvme/lv_2... 2021-11-20T05:31:52.122 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-11-20T05:31:52.151 INFO:journalctl@ceph.osd.1.smithi058.stdout:-- Logs begin at Sat 2021-11-20 05:17:50 UTC. -- 2021-11-20T05:31:52.867 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: audit 2021-11-20T05:31:51.738659+0000 2021-11-20T05:31:52.867 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: mon.smithi058 (mon.0) 254 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:52.867 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: audit 2021-11-20T05:31:51.742254+0000 mon.smithi058 (mon.0) 255 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:52.868 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: audit 2021-11-20T05:31:51.758440+0000 mon.smithi058 (mon.0) 256 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.58:9093"}]: dispatch 2021-11-20T05:31:52.868 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: audit 2021-11-20T05:31:51.758774+0000 mgr.smithi058.zagdlk (mgr.14164) 91 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.58:9093"}]: dispatch 2021-11-20T05:31:52.868 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: audit 2021-11-20T05:31:51.762722+0000 mon.smithi058 (mon.0) 257 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:52.868 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: audit 2021-11-20T05:31:51.766621+0000 mon.smithi058 (mon.0) 258 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-20T05:31:52.868 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: audit 2021-11-20T 2021-11-20T05:31:52.869 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: 05:31:51.767191+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:31:52.869 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: 92 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-20T05:31:52.869 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: audit 2021-11-20T05:31:52.869 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: 2021-11-20T05:31: 2021-11-20T05:31:52.870 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: 51.773463+0000 mon.smithi058 (mon.0) 259 : 2021-11-20T05:31:52.870 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.58:3000"}]: dispatch 2021-11-20T05:31:52.870 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: audit 2021-11- 2021-11-20T05:31:52.870 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: 20T05:31:51.773852+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:31:52.871 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: 14164) 93 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.58:3000"}]: dispatch 2021-11-20T05:31:52.871 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: audit 2021-11-20T05:31:52.871 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: 2021-11-20T05:31:51.777909+0000 2021-11-20T05:31:52.871 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: mon.smithi058 (mon.0) 260 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:52.871 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: 2021-11-20T05:31:52.872 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: audit 2021-11-20 2021-11-20T05:31:52.872 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: T05:31:51.781347+0000 mon.smithi058 (mon.0) 261 : audit 2021-11-20T05:31:52.872 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-20T05:31:52.872 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:31:52.873 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: 11-20T05:31:51.781830+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:31:52.873 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: 94 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-20T05:31:52.873 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: audit 2021-11-20T05:31:52.873 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: 2021-11-20T05:31 2021-11-20T05:31:52.873 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: :51.787425+0000 mon.smithi058 (mon.0) 2021-11-20T05:31:52.874 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: 262 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.58:9095"}]: dispatch 2021-11-20T05:31:52.874 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: audit 2021 2021-11-20T05:31:52.874 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: -11-20T05:31:51.787801+0000 2021-11-20T05:31:52.874 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 95 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.58:9095"}]: dispatch 2021-11-20T05:31:52.875 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: 2021-11-20T05:31:52.875 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: audit 2021-11-20T05:31:51 2021-11-20T05:31:52.875 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: .792051+0000 mon.smithi058 (mon.0) 263 : audit 2021-11-20T05:31:52.875 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:52.875 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: audit 2021-11-20T05 2021-11-20T05:31:52.876 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: :31:51.795570+0000 mon.smithi058 (mon.0 2021-11-20T05:31:52.876 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: ) 264 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:31:52.876 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: cluster 2021- 2021-11-20T05:31:52.876 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: 11-20T05:31:52.185027+0000 mgr.smithi058.zagdlk 2021-11-20T05:31:52.877 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:52 smithi058 conmon[33537]: (mgr.14164) 96 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:31:53.198 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit 2021-11-20T05:31:51.738659+0000 mon.smithi058 (mon.0 2021-11-20T05:31:53.198 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: ) 254 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:53.198 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit 2021-11-20T05:31:51.742254+0000 mon.smithi058 (mon.0) 255 : audit [INF] 2021-11-20T05:31:53.199 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:53.200 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit 2021- 2021-11-20T05:31:53.201 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: 11-20T05:31:51.758440+0000 2021-11-20T05:31:53.201 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: mon.smithi058 (mon.0) 256 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.58:9093"}]: dispatch 2021-11-20T05:31:53.201 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: 2021-11-20T05:31:53.201 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit 2021-11-20 2021-11-20T05:31:53.202 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: T05:31:51.758774+0000 2021-11-20T05:31:53.202 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: mgr.smithi058.zagdlk (mgr.14164) 91 : audit 2021-11-20T05:31:53.202 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.58:9093"}]: dispatch 2021-11-20T05:31:53.203 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit 2021 2021-11-20T05:31:53.203 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: -11-20T05:31: 2021-11-20T05:31:53.203 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: 51.762722+0000 mon.smithi058 (mon.0) 257 2021-11-20T05:31:53.203 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:53.204 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit 2021 2021-11-20T05:31:53.204 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: -11-20T05:31 2021-11-20T05:31:53.204 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: :51.766621+0000 mon.smithi058 (mon.0) 2021-11-20T05:31:53.204 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: 258 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-20T05:31:53.205 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit 2021- 2021-11-20T05:31:53.205 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: 11-20T05:31:51 2021-11-20T05:31:53.205 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: .767191+0000 mgr.smithi058.zagdlk (mgr.14164) 92 : 2021-11-20T05:31:53.205 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-20T05:31:53.205 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit 2021-11 2021-11-20T05:31:53.206 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: -20T05:31:51. 2021-11-20T05:31:53.206 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: 773463+0000 mon.smithi058 (mon.0) 259 : audit 2021-11-20T05:31:53.206 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.58:3000"}]: dispatch 2021-11-20T05:31:53.207 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit 2021-11 2021-11-20T05:31:53.207 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: -20T05:31:51.773852 2021-11-20T05:31:53.207 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: +0000 mgr.smithi058.zagdlk (mgr.14164) 93 : 2021-11-20T05:31:53.207 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.58:3000"}]: dispatch 2021-11-20T05:31:53.208 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit 2021-11- 2021-11-20T05:31:53.208 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: 20T05:31:51.777909+0000 2021-11-20T05:31:53.208 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: mon.smithi058 (mon.0) 260 : audit [INF] 2021-11-20T05:31:53.209 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:53.209 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit 2021-11-20T 2021-11-20T05:31:53.209 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: 05:31:51.781347+0000 mon.smithi058 (mon 2021-11-20T05:31:53.209 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: .0) 261 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-20T05:31:53.210 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit 2021-11-20T05:31:53.210 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: 2021-11-20T05:31 2021-11-20T05:31:53.210 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: :51.781830+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:31:53.210 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: 14164) 94 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-20T05:31:53.211 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit 2021 2021-11-20T05:31:53.211 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: -11-20T05:31:51 2021-11-20T05:31:53.211 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: .787425+0000 mon.smithi058 (mon.0) 262 : audit 2021-11-20T05:31:53.211 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.58:9095"}]: dispatch 2021-11-20T05:31:53.211 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit 2021-11- 2021-11-20T05:31:53.212 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: 20T05:31:51.787801+0000 2021-11-20T05:31:53.212 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: mgr.smithi058.zagdlk (mgr.14164) 95 : audit [DBG] 2021-11-20T05:31:53.212 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.58:9095"}]: dispatch 2021-11-20T05:31:53.212 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit 2021-11-20 2021-11-20T05:31:53.213 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: T05:31:51.792051+0000 mon.smithi058 2021-11-20T05:31:53.213 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: (mon.0) 263 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:53.213 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: audit 2021-11-20T05:31:53.213 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: 2021-11-20T05:31 2021-11-20T05:31:53.213 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: :51.795570+0000 mon.smithi058 (mon.0) 2021-11-20T05:31:53.214 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: 264 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:31:53.214 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: cluster 2021 2021-11-20T05:31:53.214 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: -11-20T05:31: 2021-11-20T05:31:53.214 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: 52.185027+0000 mgr.smithi058.zagdlk (mgr.14164) 96 2021-11-20T05:31:53.215 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:52 smithi187 conmon[36358]: : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:31:54.377 INFO:teuthology.orchestra.run.smithi058.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-11-20T05:31:54.377 INFO:teuthology.orchestra.run.smithi058.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-11-20T05:31:54.378 INFO:teuthology.orchestra.run.smithi058.stderr: stderr: 10+0 records in 2021-11-20T05:31:54.378 INFO:teuthology.orchestra.run.smithi058.stderr:10+0 records out 2021-11-20T05:31:54.378 INFO:teuthology.orchestra.run.smithi058.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.010101 s, 1.0 GB/s 2021-11-20T05:31:54.379 INFO:teuthology.orchestra.run.smithi058.stderr:--> Zapping successful for: 2021-11-20T05:31:54.882 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi058:vg_nvme/lv_2 2021-11-20T05:31:55.197 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:54 smithi187 conmon[36358]: audit 2021-11-20T05:31:53.943034+0000 mon.smithi058 (mon.0) 265 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:55.197 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:54 smithi187 conmon[36358]: cluster 2021-11-20T05:31:54.185276+0000 mgr.smithi058.zagdlk (mgr.14164) 97 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:31:55.219 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:54 smithi058 conmon[33537]: audit 2021-11-20T05:31:53.943034+0000 mon.smithi058 (mon.0) 265 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:55.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:54 smithi058 conmon[33537]: cluster 2021-11-20T05:31:54.185276+0000 mgr.smithi058.zagdlk (mgr.14164) 97 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:31:55.690 INFO:journalctl@ceph.osd.1.smithi058.stdout:Nov 20 05:31:55 smithi058 conmon[55656]: debug 2021-11-20T05:31:55.645+0000 7faf91cfd080 -1 osd.1 0 log_to_monitors {default=true} 2021-11-20T05:31:56.198 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:55 smithi187 conmon[36358]: audit 2021-11-20T05:31:55.649454+0000 mon.smithi058 (mon.0) 266 : audit [INF] from='osd.1 [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-11-20T05:31:56.219 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:55 smithi058 conmon[33537]: audit 2021-11-20T05:31:55.649454+0000 mon.smithi058 (mon.0) 266 : audit [INF] from='osd.1 [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-11-20T05:31:56.911 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.909+0000 7fdc1e8a5700 1 -- 172.21.15.58:0/566772482 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdc180f55c0 msgr2=0x7fdc180f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:56.911 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.909+0000 7fdc1e8a5700 1 --2- 172.21.15.58:0/566772482 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdc180f55c0 0x7fdc180f59e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fdc08004660 tx=0x7fdc08009b30).stop 2021-11-20T05:31:56.911 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.909+0000 7fdc1e8a5700 1 -- 172.21.15.58:0/566772482 shutdown_connections 2021-11-20T05:31:56.912 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.909+0000 7fdc1e8a5700 1 --2- 172.21.15.58:0/566772482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc180f6860 0x7fdc180f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:31:56.912 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.909+0000 7fdc1e8a5700 1 --2- 172.21.15.58:0/566772482 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdc180f55c0 0x7fdc180f59e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:56.912 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.909+0000 7fdc1e8a5700 1 -- 172.21.15.58:0/566772482 >> 172.21.15.58:0/566772482 conn(0x7fdc180f0d40 msgr2=0x7fdc180f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:31:56.912 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.909+0000 7fdc1e8a5700 1 -- 172.21.15.58:0/566772482 shutdown_connections 2021-11-20T05:31:56.913 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.909+0000 7fdc1e8a5700 1 -- 172.21.15.58:0/566772482 wait complete. 2021-11-20T05:31:56.913 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.909+0000 7fdc1e8a5700 1 Processor -- start 2021-11-20T05:31:56.913 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.910+0000 7fdc1e8a5700 1 -- start start 2021-11-20T05:31:56.914 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.910+0000 7fdc1e8a5700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdc180f55c0 0x7fdc1810d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:31:56.914 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.910+0000 7fdc1e8a5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc180f6860 0x7fdc1810dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:31:56.914 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.910+0000 7fdc1e8a5700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1810e0f0 con 0x7fdc180f55c0 2021-11-20T05:31:56.915 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.910+0000 7fdc1e8a5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1810e9a0 con 0x7fdc180f6860 2021-11-20T05:31:56.915 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.910+0000 7fdc17fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdc180f55c0 0x7fdc1810d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:31:56.916 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.910+0000 7fdc17fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdc180f55c0 0x7fdc1810d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40070/0 (socket says 172.21.15.58:40070) 2021-11-20T05:31:56.916 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.910+0000 7fdc17fff700 1 -- 172.21.15.58:0/2399347735 learned_addr learned my addr 172.21.15.58:0/2399347735 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:31:56.916 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.910+0000 7fdc17fff700 1 -- 172.21.15.58:0/2399347735 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc180f6860 msgr2=0x7fdc1810dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:31:56.916 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.910+0000 7fdc177fe700 1 --2- 172.21.15.58:0/2399347735 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc180f6860 0x7fdc1810dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:31:56.917 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.910+0000 7fdc17fff700 1 --2- 172.21.15.58:0/2399347735 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc180f6860 0x7fdc1810dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:31:56.917 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.910+0000 7fdc17fff700 1 -- 172.21.15.58:0/2399347735 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc08005040 con 0x7fdc180f55c0 2021-11-20T05:31:56.917 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.910+0000 7fdc177fe700 1 --2- 172.21.15.58:0/2399347735 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc180f6860 0x7fdc1810dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:31:56.917 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.910+0000 7fdc17fff700 1 --2- 172.21.15.58:0/2399347735 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdc180f55c0 0x7fdc1810d5c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fdc08009f50 tx=0x7fdc08009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:31:56.918 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.911+0000 7fdc14ff9700 1 -- 172.21.15.58:0/2399347735 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc08018070 con 0x7fdc180f55c0 2021-11-20T05:31:56.918 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.911+0000 7fdc14ff9700 1 -- 172.21.15.58:0/2399347735 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc08013410 con 0x7fdc180f55c0 2021-11-20T05:31:56.918 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.911+0000 7fdc14ff9700 1 -- 172.21.15.58:0/2399347735 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc0801c620 con 0x7fdc180f55c0 2021-11-20T05:31:56.918 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.911+0000 7fdc1e8a5700 1 -- 172.21.15.58:0/2399347735 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1810ec20 con 0x7fdc180f55c0 2021-11-20T05:31:56.918 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.911+0000 7fdc1e8a5700 1 -- 172.21.15.58:0/2399347735 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1810f130 con 0x7fdc180f55c0 2021-11-20T05:31:56.920 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.912+0000 7fdc1e8a5700 1 -- 172.21.15.58:0/2399347735 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc180f49b0 con 0x7fdc180f55c0 2021-11-20T05:31:56.920 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.912+0000 7fdc14ff9700 1 -- 172.21.15.58:0/2399347735 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fdc0801d420 con 0x7fdc180f55c0 2021-11-20T05:31:56.921 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.913+0000 7fdc14ff9700 1 --2- 172.21.15.58:0/2399347735 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdc0005c2b0 0x7fdc0005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:31:56.921 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.913+0000 7fdc14ff9700 1 -- 172.21.15.58:0/2399347735 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2018+0+0 (secure 0 0 0) 0x7fdc08077060 con 0x7fdc180f55c0 2021-11-20T05:31:56.921 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.913+0000 7fdc177fe700 1 --2- 172.21.15.58:0/2399347735 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdc0005c2b0 0x7fdc0005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:31:56.921 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.913+0000 7fdc177fe700 1 --2- 172.21.15.58:0/2399347735 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdc0005c2b0 0x7fdc0005e770 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fdc0c00a7e0 tx=0x7fdc0c008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:31:56.922 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:56.917+0000 7fdc14ff9700 1 -- 172.21.15.58:0/2399347735 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc0804aa40 con 0x7fdc180f55c0 2021-11-20T05:31:57.109 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:31:57.110+0000 7fdc1e8a5700 1 -- 172.21.15.58:0/2399347735 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi058:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc180fb2c0 con 0x7fdc0005c2b0 2021-11-20T05:31:57.219 INFO:journalctl@ceph.osd.1.smithi058.stdout:Nov 20 05:31:56 smithi058 conmon[55656]: debug 2021-11-20T05:31:56.955+0000 7faf8826e700 -1 osd.1 0 waiting for initial osdmap 2021-11-20T05:31:57.220 INFO:journalctl@ceph.osd.1.smithi058.stdout:Nov 20 05:31:56 smithi058 conmon[55656]: debug 2021-11-20T05:31:56.961+0000 7faf821e5700 -1 osd.1 13 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2021-11-20T05:31:57.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:56 smithi058 conmon[33537]: audit 2021-11-20T05:31:55 2021-11-20T05:31:57.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:56 smithi058 conmon[33537]: .951649+0000 mon.smithi058 (mon.0) 267 : audit [INF] from='osd.1 [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-11-20T05:31:57.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:56 smithi058 conmon[33537]: cluster 2021-11-20T05:31:55.951699+0000 mon.smithi058 (mon.0) 268 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-11-20T05:31:57.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:56 smithi058 conmon[33537]: audit 2021-11-20T05:31:55.951808+0000 mon.smithi058 (mon.0) 269 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T05:31:57.223 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:56 smithi058 conmon[33537]: audit 2021-11-20T05:31:55.952154+0000 mon.smithi058 (mon.0) 270 : audit [INF] from='osd.1 [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi058", "root=default"]}]: dispatch 2021-11-20T05:31:57.224 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:56 smithi058 conmon[33537]: cluster 2021-11-20T05:31:56.185497+0000 mgr.smithi058.zagdlk (mgr.14164) 98 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:31:57.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:56 smithi187 conmon[36358]: audit 2021-11-20T05:31:55.951649+0000 mon.smithi058 (mon.0) 267 : audit [INF] from='osd.1 [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-11-20T05:31:57.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:56 smithi187 conmon[36358]: cluster 2021- 2021-11-20T05:31:57.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:56 smithi187 conmon[36358]: 11-20T05:31:55.951699+0000 mon.smithi058 (mon.0) 268 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-11-20T05:31:57.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:56 smithi187 conmon[36358]: audit 2021-11-20T05:31:55.951808+0000 mon.smithi058 (mon.0) 269 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T05:31:57.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:56 smithi187 conmon[36358]: audit 2021 2021-11-20T05:31:57.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:56 smithi187 conmon[36358]: -11-20T05:31:55.952154+0000 mon.smithi058 (mon.0) 270 : audit [INF] from='osd.1 [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi058", "root=default"]}]: dispatch 2021-11-20T05:31:57.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:56 smithi187 conmon[36358]: cluster 2021-11-20T05:31:56.185497+0000 mgr.smithi058.zagdlk 2021-11-20T05:31:57.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:56 smithi187 conmon[36358]: (mgr.14164) 98 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:31:58.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:57 smithi058 conmon[33537]: audit 2021-11-20T05:31:56.953441+0000 mon.smithi058 (mon.0) 271 : audit [INF] from='osd.1 [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi058", "root=default"]}]': finished 2021-11-20T05:31:58.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:57 smithi058 conmon[33537]: cluster 2021-11-20T05:31:56.953497+0000 mon.smithi058 (mon.0) 272 : cluster [DBG] osdmap e13: 2 total, 1 up, 2 in 2021-11-20T05:31:58.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:57 smithi058 conmon[33537]: audit 2021-11-20T05:31:56.953804+0000 mon.smithi058 (mon.0) 273 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T05:31:58.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:57 smithi058 conmon[33537]: audit 2021-11-20T05:31:56.958479+0000 mon.smithi058 (mon.0) 274 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T05:31:58.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:57 smithi058 conmon[33537]: audit 2021-11-20T05:31:58.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:57 smithi058 conmon[33537]: 2021-11-20T05:31:57.111771+0000 mgr.smithi058.zagdlk (mgr.14164) 99 : audit [DBG] from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi058:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:31:58.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:57 smithi058 conmon[33537]: audit 2021-11-20T05:31:57.112556+0000 mon.smithi058 (mon.0) 275 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T05:31:58.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:57 smithi058 conmon[33537]: cephadm 2021-11-20T05:31:57.113794+0000 mgr.smithi058.zagdlk (mgr.14164) 100 : cephadm [INF] Applying service osd.None on host smithi058... 2021-11-20T05:31:58.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:57 smithi058 conmon[33537]: audit 2021-11-20T05:31:57.114016+0000 mon.smithi058 (mon.0) 276 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T05:31:58.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:57 smithi058 conmon[33537]: audit 2021-11-20T05:31:57.114645+0000 mon.smithi058 (mon.0) 277 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:58.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:57 smithi058 conmon[33537]: audit 2021-11-20T05:31:57.461111+0000 mon.smithi058 (mon.0) 278 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:58.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: audit 2021-11-20T05:31:56.953441+0000 mon.smithi058 (mon.0) 271 : audit [INF] from='osd.1 [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi058", "root=default"]}]': finished 2021-11-20T05:31:58.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: cluster 2021-11-20T05:31:56.953497+0000 mon.smithi058 (mon.0) 272 : cluster [DBG] osdmap e13: 2 total, 1 up, 2 in 2021-11-20T05:31:58.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: audit 2021-11-20T05:31:56.953804+0000 mon.smithi058 (mon.0) 273 : 2021-11-20T05:31:58.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T05:31:58.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: audit 2021- 2021-11-20T05:31:58.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: 11-20T05:31:56.958479+0000 mon.smithi058 (mon.0) 274 : audit [DBG] 2021-11-20T05:31:58.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T05:31:58.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: audit 2021-11-20T05:31:58.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: 2021-11-20T05:31:57.111771+0000 mgr.smithi058.zagdlk (mgr.14164) 99 : audit 2021-11-20T05:31:58.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: [DBG] from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi058:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:31:58.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: audit 2021-11-20T05:31:58.451 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: 2021-11-20T05:31:57.112556+0000 mon.smithi058 (mon.0) 275 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T05:31:58.451 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: cephadm 2021-11-20T 2021-11-20T05:31:58.451 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: 05:31:57.113794+0000 mgr.smithi058.zagdlk (mgr.14164) 100 : cephadm [INF] Applying service osd.None on host smithi058... 2021-11-20T05:31:58.451 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: audit 2021-11-20 2021-11-20T05:31:58.452 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: T05:31:57.114016+0000 mon.smithi058 (mon.0) 276 : audit 2021-11-20T05:31:58.452 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T05:31:58.452 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: audit 2021-11-20 2021-11-20T05:31:58.452 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: T05:31:57.114645+0000 mon.smithi058 (mon.0) 277 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:31:58.453 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: audit 2021-11-20T05:31:57.461111+0000 2021-11-20T05:31:58.453 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:57 smithi187 conmon[36358]: mon.smithi058 (mon.0) 278 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:31:59.219 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:58 smithi058 conmon[33537]: cluster 2021-11-20T05:31:56.685702+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T05:31:59.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:58 smithi058 conmon[33537]: cluster 2021-11-20T05:31:59.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:58 smithi058 conmon[33537]: 2021-11-20T05:31:56.685869+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T05:31:59.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:58 smithi058 conmon[33537]: cluster 2021-11-20T05:31:57.958431+0000 mon.smithi058 (mon.0) 279 : cluster [INF] osd.1 [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869] boot 2021-11-20T05:31:59.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:58 smithi058 conmon[33537]: cluster 2021-11-20T05:31:57.958474+0000 mon.smithi058 (mon.0) 280 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-11-20T05:31:59.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:58 smithi058 conmon[33537]: audit 2021-11-20T05:31:57.958754+0000 mon.smithi058 (mon.0) 281 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T05:31:59.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:58 smithi058 conmon[33537]: cluster 2021-11-20T05:31:58.185836+0000 mgr.smithi058.zagdlk (mgr.14164) 101 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:31:59.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:58 smithi187 conmon[36358]: cluster 2021-11-20T05:31:56.685702+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T05:31:59.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:58 smithi187 conmon[36358]: cluster 2021-11-20T05:31:56.685869+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T05:31:59.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:58 smithi187 conmon[36358]: cluster 2021-11-20T05:31:57.958431+0000 mon.smithi058 ( 2021-11-20T05:31:59.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:58 smithi187 conmon[36358]: mon.0) 279 : cluster [INF] osd.1 [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869] boot 2021-11-20T05:31:59.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:58 smithi187 conmon[36358]: cluster 2021-11-20T05:31:59.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:58 smithi187 conmon[36358]: 2021-11-20T05:31:57.958474+0000 mon.smithi058 (mon.0) 280 : cluster 2021-11-20T05:31:59.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:58 smithi187 conmon[36358]: [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-11-20T05:31:59.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:58 smithi187 conmon[36358]: audit 2021-11 2021-11-20T05:31:59.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:58 smithi187 conmon[36358]: -20T05:31:57.958754+0000 mon.smithi058 (mon.0) 281 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T05:31:59.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:58 smithi187 conmon[36358]: cluster 2021-11-20 2021-11-20T05:31:59.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:58 smithi187 conmon[36358]: T05:31:58.185836+0000 mgr.smithi058.zagdlk (mgr.14164) 101 : 2021-11-20T05:31:59.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:58 smithi187 conmon[36358]: cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T05:32:00.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:59 smithi058 conmon[33537]: cluster 2021-11-20T05:31:58.968600 2021-11-20T05:32:00.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:59 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 282 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2021-11-20T05:32:00.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:59 smithi058 conmon[33537]: audit 2021-11-20T05:31:59.438594+0000 mon.smithi058 (mon.0) 283 : audit [INF] from='client.? 172.21.15.58:0/867041082' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b35ec54-f60f-49c5-ab1b-d2c5b5eb72e9"}]: dispatch 2021-11-20T05:32:00.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:59 smithi058 conmon[33537]: audit 2021-11-20T05:31:59.442324+0000 mon.smithi058 (mon.0) 284 : audit [INF] from='client.? 172.21.15.58:0/867041082' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b35ec54-f60f-49c5-ab1b-d2c5b5eb72e9"}]': finished 2021-11-20T05:32:00.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:59 smithi058 conmon[33537]: cluster 2021-11-20T05:31:59.442368+0000 mon.smithi058 (mon.0) 285 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-11-20T05:32:00.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:59 smithi058 conmon[33537]: audit 2021-11-20T05:31:59.442450+0000 mon.smithi058 (mon.0) 286 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T05:32:00.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:31:59 smithi058 conmon[33537]: audit 2021-11-20T05:31:59.876714+0000 mon.smithi058 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.58:0/2913721977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T05:32:00.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:59 smithi187 conmon[36358]: cluster 2021-11-20T05:31:58.968600+0000 mon.smithi058 (mon.0) 282 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2021-11-20T05:32:00.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:59 smithi187 conmon[36358]: audit 2021-11-20T05:31:59.438594+0000 mon.smithi058 (mon.0) 283 : audit [INF] from='client.? 172.21.15.58:0/867041082' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b35ec54-f60f-49c5-ab1b-d2c5b5eb72e9"}]: dispatch 2021-11-20T05:32:00.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:59 smithi187 conmon[36358]: audit 2021-11-20T05:31:59.442324+0000 mon.smithi058 (mon 2021-11-20T05:32:00.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:59 smithi187 conmon[36358]: .0) 284 : audit [INF] from='client.? 172.21.15.58:0/867041082' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b35ec54-f60f-49c5-ab1b-d2c5b5eb72e9"}]': finished 2021-11-20T05:32:00.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:59 smithi187 conmon[36358]: cluster 2021-11-20T05 2021-11-20T05:32:00.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:59 smithi187 conmon[36358]: :31:59.442368+0000 mon.smithi058 (mon.0) 285 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-11-20T05:32:00.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:59 smithi187 conmon[36358]: audit 2021-11-20T05:31:59 2021-11-20T05:32:00.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:59 smithi187 conmon[36358]: .442450+0000 mon.smithi058 (mon.0) 286 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T05:32:00.451 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:59 smithi187 conmon[36358]: audit 2021-11- 2021-11-20T05:32:00.451 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:59 smithi187 conmon[36358]: 20T05:31:59.876714+0000 mon.smithi058 (mon.0) 287 2021-11-20T05:32:00.452 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:31:59 smithi187 conmon[36358]: : audit [DBG] from='client.? 172.21.15.58:0/2913721977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T05:32:01.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:00 smithi187 conmon[36358]: cluster 2021-11-20T05:32:00.186237+0000 mgr.smithi058.zagdlk (mgr.14164) 102 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:01.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:00 smithi187 conmon[36358]: audit 2021-11-20T05:32:00.356178+0000 mon.smithi058 (mon.0) 288 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:32:01.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:00 smithi187 conmon[36358]: audit 2021-11-20T05:32:00.363688+0000 mon.smithi058 (mon.0) 289 : audit 2021-11-20T05:32:01.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:00 smithi187 conmon[36358]: [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:32:01.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:00 smithi058 conmon[33537]: cluster 2021-11-20T05:32:00.186237+0000 mgr.smithi058.zagdlk (mgr.14164) 102 2021-11-20T05:32:01.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:00 smithi058 conmon[33537]: : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:01.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:00 smithi058 conmon[33537]: audit 2021-11-20T05:32:00.356178+0000 mon.smithi058 (mon.0) 288 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:32:01.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:00 smithi058 conmon[33537]: audit 2021-11-20T05: 2021-11-20T05:32:01.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:00 smithi058 conmon[33537]: 32:00.363688+0000 mon.smithi058 (mon.0) 289 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:32:03.558 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:03 smithi058 conmon[33537]: cluster 2021-11-20T05:32:02.186572+0000 mgr.smithi058.zagdlk (mgr.14164) 103 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:03.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:03 smithi187 conmon[36358]: cluster 2021-11-20T05:32:02.186572+0000 mgr.smithi058.zagdlk (mgr.14164) 103 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:03.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:03 smithi187 conmon[36358]: 2021-11-20T05:32:05.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:05 smithi187 conmon[36358]: cluster 2021-11-20T05:32:04.186907+0000 mgr.smithi058.zagdlk (mgr.14164) 104 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:05.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:05 smithi058 conmon[33537]: cluster 2021-11-20T05:32:04.186907+0000 mgr.smithi058.zagdlk (mgr.14164) 104 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:06.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:06 smithi187 conmon[36358]: cephadm 2021-11-20T05:32:05.504345+0000 mgr.smithi058.zagdlk (mgr.14164) 105 : cephadm [INF] Detected new or changed devices on smithi058 2021-11-20T05:32:06.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:06 smithi187 conmon[36358]: audit 2021-11-20T05:32:05.509316+0000 mon.smithi058 (mon.0) 290 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:06.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:06 smithi187 conmon[36358]: audit 2021-11-20T05:32:05.512629+0000 mon.smithi058 (mon.0) 291 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:06.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:06 smithi187 conmon[36358]: audit 2021-11-20T05:32:05.515464+0000 mon.smithi058 (mon.0) 292 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:32:06.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:06 smithi187 conmon[36358]: cluster 2021-11-20T05:32:06.187218+0000 mgr.smithi058.zagdlk (mgr.14164) 106 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:06.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:06 smithi187 conmon[36358]: audit 2021-11-20T05:32:06.215589+0000 mon.smithi058 (mon.0) 293 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-11-20T05:32:06.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:06 smithi187 conmon[36358]: audit 2021-11-20T05:32:06.216314+0000 mon.smithi058 (mon.0) 294 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:32:06.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:06 smithi187 conmon[36358]: cephadm 2021-11-20T05:32:06.216878+0000 mgr.smithi058.zagdlk (mgr.14164) 107 : cephadm 2021-11-20T05:32:06.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:06 smithi187 conmon[36358]: [INF] Deploying daemon osd.2 on smithi058 2021-11-20T05:32:06.969 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:06 smithi058 conmon[33537]: cephadm 2021-11-20T05:32:05.504345+0000 mgr.smithi058.zagdlk (mgr.14164) 105 : cephadm [INF] Detected new or changed devices on smithi058 2021-11-20T05:32:06.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:06 smithi058 conmon[33537]: audit 2021-11-20T05:32:05.509316+0000 mon.smithi058 (mon.0) 290 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:06.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:06 smithi058 conmon[33537]: audit 2021-11-20T05:32:05.512629+0000 mon.smithi058 (mon.0) 291 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:06.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:06 smithi058 conmon[33537]: audit 2021-11-20T05:32:05.515464+0000 mon.smithi058 (mon.0) 292 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:32:06.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:06 smithi058 conmon[33537]: cluster 2021-11-20T05:32:06.187218+0000 mgr.smithi058.zagdlk (mgr.14164) 106 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:06.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:06 smithi058 conmon[33537]: audit 2021-11-20T05:32:06.215589+0000 mon.smithi058 (mon.0) 293 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-11-20T05:32:06.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:06 smithi058 conmon[33537]: audit 2021-11-20T05:32:06.216314+0000 mon.smithi058 (mon.0) 294 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:32:06.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:06 smithi058 conmon[33537]: cephadm 2021-11-20T05:32:06.216878+0000 mgr.smithi058.zagdlk (mgr.14164) 107 : cephadm [INF] Deploying daemon osd.2 on smithi058 2021-11-20T05:32:08.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:08 smithi187 conmon[36358]: audit 2021-11-20T05:32:07.662362+0000 mon.smithi058 (mon.0) 295 : audit [INF] 2021-11-20T05:32:08.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:08 smithi187 conmon[36358]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:08.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:08 smithi187 conmon[36358]: cluster 2021-11-20T05:32:08.187476+0000 mgr.smithi058.zagdlk (mgr.14164) 108 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:08.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:08 smithi058 conmon[33537]: audit 2021-11-20T05:32:07.662362+0000 mon.smithi058 (mon.0) 295 : audit [INF] 2021-11-20T05:32:08.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:08 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:08.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:08 smithi058 conmon[33537]: cluster 2021-11-20T05:32:08.187476+0000 mgr.smithi058.zagdlk (mgr.14164) 108 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:10.276 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:32:10.276+0000 7fdc14ff9700 1 -- 172.21.15.58:0/2399347735 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fdc180fb2c0 con 0x7fdc0005c2b0 2021-11-20T05:32:10.276 INFO:teuthology.orchestra.run.smithi058.stdout:Created osd(s) 2 on host 'smithi058' 2021-11-20T05:32:10.279 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:32:10.279+0000 7fdc1e8a5700 1 -- 172.21.15.58:0/2399347735 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdc0005c2b0 msgr2=0x7fdc0005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:32:10.279 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:32:10.279+0000 7fdc1e8a5700 1 --2- 172.21.15.58:0/2399347735 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdc0005c2b0 0x7fdc0005e770 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fdc0c00a7e0 tx=0x7fdc0c008040).stop 2021-11-20T05:32:10.280 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:32:10.279+0000 7fdc1e8a5700 1 -- 172.21.15.58:0/2399347735 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdc180f55c0 msgr2=0x7fdc1810d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:32:10.280 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:32:10.279+0000 7fdc1e8a5700 1 --2- 172.21.15.58:0/2399347735 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdc180f55c0 0x7fdc1810d5c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fdc08009f50 tx=0x7fdc08009b30).stop 2021-11-20T05:32:10.280 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:32:10.279+0000 7fdc1e8a5700 1 -- 172.21.15.58:0/2399347735 shutdown_connections 2021-11-20T05:32:10.280 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:32:10.279+0000 7fdc1e8a5700 1 --2- 172.21.15.58:0/2399347735 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdc0005c2b0 0x7fdc0005e770 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:10.280 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:32:10.279+0000 7fdc1e8a5700 1 --2- 172.21.15.58:0/2399347735 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdc180f55c0 0x7fdc1810d5c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:10.281 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:32:10.279+0000 7fdc1e8a5700 1 --2- 172.21.15.58:0/2399347735 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc180f6860 0x7fdc1810dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:10.281 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:32:10.279+0000 7fdc1e8a5700 1 -- 172.21.15.58:0/2399347735 >> 172.21.15.58:0/2399347735 conn(0x7fdc180f0d40 msgr2=0x7fdc1805f620 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:32:10.281 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:32:10.280+0000 7fdc1e8a5700 1 -- 172.21.15.58:0/2399347735 shutdown_connections 2021-11-20T05:32:10.281 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:32:10.280+0000 7fdc1e8a5700 1 -- 172.21.15.58:0/2399347735 wait complete. 2021-11-20T05:32:10.798 DEBUG:teuthology.orchestra.run.smithi058:osd.2> sudo journalctl -f -n 0 -u ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.2.service 2021-11-20T05:32:10.802 INFO:tasks.cephadm:Deploying osd.3 on smithi187 with /dev/vg_nvme/lv_4... 2021-11-20T05:32:10.802 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-11-20T05:32:10.832 INFO:journalctl@ceph.osd.2.smithi058.stdout:-- Logs begin at Sat 2021-11-20 05:17:50 UTC. -- 2021-11-20T05:32:11.528 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:11 smithi187 conmon[36358]: cluster 2021-11-20T05:32:10.187987+0000 mgr.smithi058.zagdlk (mgr.14164) 109 2021-11-20T05:32:11.529 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:11 smithi187 conmon[36358]: : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:11.529 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:11 smithi187 conmon[36358]: audit 2021-11-20T05:32:10.272388+0000 mon.smithi058 (mon.0) 296 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:11.529 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:11 smithi187 conmon[36358]: audit 2021-11-20T05:32:10.276382+0000 mon.smithi058 (mon.0) 297 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:11.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:11 smithi058 conmon[33537]: cluster 2021-11-20T05:32:10.187987+0000 2021-11-20T05:32:11.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:11 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 109 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:11.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:11 smithi058 conmon[33537]: audit 2021-11-20T05:32:10.272388+0000 mon.smithi058 (mon.0) 296 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:11.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:11 smithi058 conmon[33537]: audit 2021-11-20T05:32:10.276382+0000 mon.smithi058 (mon.0) 297 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:12.686 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-11-20T05:32:12.687 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-11-20T05:32:12.687 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2021-11-20T05:32:12.687 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2021-11-20T05:32:12.687 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120421 s, 871 MB/s 2021-11-20T05:32:12.688 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2021-11-20T05:32:12.930 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:12 smithi058 conmon[33537]: audit 2021-11-20T05:32:11.666367+0000 mon.smithi058 (mon.0) 298 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:12.930 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:12 smithi058 conmon[33537]: cluster 2021-11-20T05:32:12.188264+0000 mgr.smithi058.zagdlk (mgr.14164) 110 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:12.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:12 smithi187 conmon[36358]: audit 2021-11-20T05:32:11.666367+0000 mon.smithi058 (mon.0) 298 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:12.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:12 smithi187 conmon[36358]: cluster 2021-11-20T05:32:12.188264+0000 mgr.smithi058.zagdlk (mgr.14164) 110 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:13.114 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi187:vg_nvme/lv_4 2021-11-20T05:32:14.469 INFO:journalctl@ceph.osd.2.smithi058.stdout:Nov 20 05:32:14 smithi058 conmon[61992]: debug 2021-11-20T05:32:14.214+0000 7f91bc614080 -1 osd.2 0 log_to_monitors {default=true} 2021-11-20T05:32:14.624 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.622+0000 7f9bbb220700 1 -- 172.21.15.187:0/3510206184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bb406a350 msgr2=0x7f9bb406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:32:14.625 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.622+0000 7f9bbb220700 1 --2- 172.21.15.187:0/3510206184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bb406a350 0x7f9bb406a7d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f9ba0004660 tx=0x7f9ba0009b30).stop 2021-11-20T05:32:14.625 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.622+0000 7f9bbb220700 1 -- 172.21.15.187:0/3510206184 shutdown_connections 2021-11-20T05:32:14.625 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.622+0000 7f9bbb220700 1 --2- 172.21.15.187:0/3510206184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bb406a350 0x7f9bb406a7d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:14.625 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.622+0000 7f9bbb220700 1 --2- 172.21.15.187:0/3510206184 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9bb406b960 0x7f9bb4069e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:14.626 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.622+0000 7f9bbb220700 1 -- 172.21.15.187:0/3510206184 >> 172.21.15.187:0/3510206184 conn(0x7f9bb40f1e40 msgr2=0x7f9bb40f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:32:14.626 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.622+0000 7f9bbb220700 1 -- 172.21.15.187:0/3510206184 shutdown_connections 2021-11-20T05:32:14.626 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.622+0000 7f9bbb220700 1 -- 172.21.15.187:0/3510206184 wait complete. 2021-11-20T05:32:14.626 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.623+0000 7f9bbb220700 1 Processor -- start 2021-11-20T05:32:14.627 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.623+0000 7f9bbb220700 1 -- start start 2021-11-20T05:32:14.627 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.623+0000 7f9bbb220700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bb406a350 0x7f9bb4109120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:32:14.627 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.623+0000 7f9bbb220700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9bb406b960 0x7f9bb4109610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:32:14.627 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.623+0000 7f9bbb220700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb4109bc0 con 0x7f9bb406b960 2021-11-20T05:32:14.628 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.623+0000 7f9bbb220700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb4109d00 con 0x7f9bb406a350 2021-11-20T05:32:14.628 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.623+0000 7f9bb8fbc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bb406a350 0x7f9bb4109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:32:14.628 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.623+0000 7f9bb8fbc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bb406a350 0x7f9bb4109120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44342/0 (socket says 172.21.15.187:44342) 2021-11-20T05:32:14.628 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.623+0000 7f9bb8fbc700 1 -- 172.21.15.187:0/163777562 learned_addr learned my addr 172.21.15.187:0/163777562 (peer_addr_for_me v2:172.21.15.187:0/0) 2021-11-20T05:32:14.628 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.623+0000 7f9bb3fff700 1 --2- 172.21.15.187:0/163777562 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9bb406b960 0x7f9bb4109610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:32:14.629 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.623+0000 7f9bb8fbc700 1 -- 172.21.15.187:0/163777562 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9bb406b960 msgr2=0x7f9bb4109610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:32:14.629 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.623+0000 7f9bb8fbc700 1 --2- 172.21.15.187:0/163777562 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9bb406b960 0x7f9bb4109610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:14.629 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.623+0000 7f9bb8fbc700 1 -- 172.21.15.187:0/163777562 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba0005040 con 0x7f9bb406a350 2021-11-20T05:32:14.629 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.623+0000 7f9bb8fbc700 1 --2- 172.21.15.187:0/163777562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bb406a350 0x7f9bb4109120 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f9ba80026d0 tx=0x7f9ba8009de0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:32:14.630 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.624+0000 7f9bb17fa700 1 -- 172.21.15.187:0/163777562 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ba80079c0 con 0x7f9bb406a350 2021-11-20T05:32:14.630 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.624+0000 7f9bbb220700 1 -- 172.21.15.187:0/163777562 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb410a6a0 con 0x7f9bb406a350 2021-11-20T05:32:14.630 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.624+0000 7f9bbb220700 1 -- 172.21.15.187:0/163777562 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb410ac60 con 0x7f9bb406a350 2021-11-20T05:32:14.630 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.624+0000 7f9bb17fa700 1 -- 172.21.15.187:0/163777562 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ba800f040 con 0x7f9bb406a350 2021-11-20T05:32:14.631 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.624+0000 7f9bb17fa700 1 -- 172.21.15.187:0/163777562 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ba80148b0 con 0x7f9bb406a350 2021-11-20T05:32:14.632 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.625+0000 7f9bbb220700 1 -- 172.21.15.187:0/163777562 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b94004fa0 con 0x7f9bb406a350 2021-11-20T05:32:14.632 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.625+0000 7f9bb17fa700 1 -- 172.21.15.187:0/163777562 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9ba8014a10 con 0x7f9bb406a350 2021-11-20T05:32:14.632 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.625+0000 7f9bb17fa700 1 --2- 172.21.15.187:0/163777562 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9ba405c2b0 0x7f9ba405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:32:14.633 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.626+0000 7f9bb17fa700 1 -- 172.21.15.187:0/163777562 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2429+0+0 (secure 0 0 0) 0x7f9ba8075160 con 0x7f9bb406a350 2021-11-20T05:32:14.633 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.626+0000 7f9bb3fff700 1 --2- 172.21.15.187:0/163777562 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9ba405c2b0 0x7f9ba405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:32:14.633 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.626+0000 7f9bb3fff700 1 --2- 172.21.15.187:0/163777562 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9ba405c2b0 0x7f9ba405e770 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f9ba0009f50 tx=0x7f9ba0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:32:14.633 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.630+0000 7f9bb17fa700 1 -- 172.21.15.187:0/163777562 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ba804c0f0 con 0x7f9bb406a350 2021-11-20T05:32:14.815 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:14.815+0000 7f9bbb220700 1 -- 172.21.15.187:0/163777562 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b94000bc0 con 0x7f9ba405c2b0 2021-11-20T05:32:15.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:15 smithi187 conmon[36358]: cluster 2021-11-20T05:32:14.188551+0000 2021-11-20T05:32:15.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:15 smithi187 conmon[36358]: mgr.smithi058.zagdlk (mgr.14164) 111 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:15.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:15 smithi187 conmon[36358]: audit 2021-11-20T05:32:14.218493+0000 mon.smithi058 (mon.0) 299 : audit [INF] from='osd.2 [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-11-20T05:32:15.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:15 smithi187 conmon[36358]: audit 2021-11-20T05:32:14.489776+0000 mon.smithi058 (mon.0) 300 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:15.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:15 smithi187 conmon[36358]: audit 2021-11-20T05:32:14.493842+0000 mon.smithi058 (mon.0) 301 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:15.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:15 smithi187 conmon[36358]: audit 2021-11-20T05:32:14.496624+0000 mon.smithi058 (mon.0) 302 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:32:15.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:15 smithi187 conmon[36358]: audit 2021-11-20T05:32:14.818561+0000 mon.smithi058 (mon.0) 303 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T05:32:15.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:15 smithi187 conmon[36358]: audit 2021-11-20T05:32:14.820233+0000 mon.smithi058 (mon.0) 304 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T05:32:15.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:15 smithi187 conmon[36358]: audit 2021-11-20T05:32:14.820898+0000 mon.smithi058 (mon.0) 305 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:32:15.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:15 smithi058 conmon[33537]: cluster 2021-11-20T05:32:14.188551+0000 mgr.smithi058.zagdlk (mgr.14164) 111 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:15.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:15 smithi058 conmon[33537]: audit 2021-11-20T05:32:14.218493+0000 mon.smithi058 (mon.0) 299 : audit [INF] from='osd.2 [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-11-20T05:32:15.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:15 smithi058 conmon[33537]: audit 2021-11-20T05:32:14.489776+0000 mon.smithi058 (mon.0) 300 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:15.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:15 smithi058 conmon[33537]: audit 2021-11-20T05:32:14.493842+0000 mon.smithi058 (mon.0) 301 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:15.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:15 smithi058 conmon[33537]: audit 2021-11-20T05:32:14.496624+0000 mon.smithi058 (mon.0) 302 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:32:15.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:15 smithi058 conmon[33537]: audit 2021-11-20T05:32:14.818561+0000 mon.smithi058 (mon.0) 303 : audit 2021-11-20T05:32:15.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:15 smithi058 conmon[33537]: [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T05:32:15.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:15 smithi058 conmon[33537]: audit 2021-11-20T05:32:14.820233+0000 mon.smithi058 (mon.0) 304 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T05:32:15.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:15 smithi058 conmon[33537]: audit 2021-11-20T05:32:14.820898+0000 mon.smithi058 (mon.0) 305 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:32:16.231 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:16 smithi187 conmon[36358]: audit 2021-11-20T05:32:14.817693+0000 2021-11-20T05:32:16.231 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:16 smithi187 conmon[36358]: mgr.smithi058.zagdlk (mgr.14164) 112 : audit [DBG] from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:32:16.232 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:16 smithi187 conmon[36358]: cephadm 2021-11-20T05:32:14.820004+0000 mgr.smithi058.zagdlk (mgr.14164) 113 : cephadm [INF] Applying service osd.None on host smithi187... 2021-11-20T05:32:16.232 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:16 smithi187 conmon[36358]: audit 2021-11-20T05:32:15.196305+0000 mon.smithi058 (mon.0) 306 : audit [INF] from='osd.2 [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-11-20T05:32:16.232 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:16 smithi187 conmon[36358]: cluster 2021-11-20T05:32:15.196394+0000 mon.smithi058 (mon.0) 307 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-11-20T05:32:16.232 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:16 smithi187 conmon[36358]: audit 2021-11-20T05:32:15.196502+0000 mon.smithi058 (mon.0) 308 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T05:32:16.232 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:16 smithi187 conmon[36358]: audit 2021-11-20T05:32:15.196867+0000 mon.smithi058 (mon.0) 309 : audit [INF] from='osd.2 [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi058", "root=default"]}]: dispatch 2021-11-20T05:32:16.469 INFO:journalctl@ceph.osd.2.smithi058.stdout:Nov 20 05:32:16 smithi058 conmon[61992]: debug 2021-11-20T05:32:16.202+0000 7f91b2b85700 -1 osd.2 0 waiting for initial osdmap 2021-11-20T05:32:16.470 INFO:journalctl@ceph.osd.2.smithi058.stdout:Nov 20 05:32:16 smithi058 conmon[61992]: debug 2021-11-20T05:32:16.207+0000 7f91ab2f9700 -1 osd.2 18 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2021-11-20T05:32:16.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:16 smithi058 conmon[33537]: audit 2021-11-20T05:32:14.817693+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:32:16.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:16 smithi058 conmon[33537]: mgr.14164) 112 : audit [DBG] from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:32:16.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:16 smithi058 conmon[33537]: cephadm 2021-11-20T05:32:14.820004+0000 mgr.smithi058.zagdlk (mgr.14164) 113 : cephadm [INF] Applying service osd.None on host smithi187... 2021-11-20T05:32:16.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:16 smithi058 conmon[33537]: audit 2021-11-20T05:32:15.196305+0000 mon.smithi058 (mon.0) 306 : audit [INF] from='osd.2 [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-11-20T05:32:16.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:16 smithi058 conmon[33537]: cluster 2021-11-20T05:32:15.196394+0000 mon.smithi058 (mon.0) 307 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-11-20T05:32:16.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:16 smithi058 conmon[33537]: audit 2021-11-20T05:32:15.196502+0000 mon.smithi058 (mon.0) 308 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T05:32:16.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:16 smithi058 conmon[33537]: audit 2021-11-20T05:32:15.196867+0000 mon.smithi058 (mon.0) 309 : audit [INF] from='osd.2 [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi058", "root=default"]}]: dispatch 2021-11-20T05:32:17.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: cluster 2021-11 2021-11-20T05:32:17.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: -20T05:32:16.188824+0000 mgr.smithi058.zagdlk (mgr.14164) 114 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:17.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: audit 2021-11-20T05:32:16.200230+0000 mon.smithi058 (mon.0) 310 : audit [INF] from='osd.2 [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi058", "root=default"]}]': finished 2021-11-20T05:32:17.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: cluster 2021-11-20T05:32:16.200291+0000 mon.smithi058 (mon.0 2021-11-20T05:32:17.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: ) 311 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2021-11-20T05:32:17.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:32:17.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: 11-20T05:32:16.200550+0000 2021-11-20T05:32:17.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: mon.smithi058 (mon.0) 312 : audit [DBG] 2021-11-20T05:32:17.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T05:32:17.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: audit 2021-11-20 2021-11-20T05:32:17.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: T05:32:16.203747+0000 2021-11-20T05:32:17.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: mon.smithi058 (mon.0) 313 : audit [DBG] 2021-11-20T05:32:17.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T05:32:17.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: audit 2021-11-20 2021-11-20T05:32:17.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: T05:32:16.905937+0000 2021-11-20T05:32:17.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: mon.smithi187 (mon.1) 2 : audit [INF] 2021-11-20T05:32:17.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: from='client.? 172.21.15.187:0/3062992476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8390f146-b7e1-4039-83d3-7ed1c832dc8e"}]: dispatch 2021-11-20T05:32:17.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:32:17.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: -20T05:32:16. 2021-11-20T05:32:17.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: 906472+0000 mon.smithi058 (mon.0) 314 : 2021-11-20T05:32:17.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8390f146-b7e1-4039-83d3-7ed1c832dc8e"}]: dispatch 2021-11-20T05:32:17.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:32:17.475 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: -11-20T05:32 2021-11-20T05:32:17.475 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: :16.910472+0000 mon.smithi058 (mon.0 2021-11-20T05:32:17.475 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: ) 315 : cluster [INF] osd.2 [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076] boot 2021-11-20T05:32:17.475 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: audit 2021-11-20T05:32:17.476 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: 2021-11-20T05:32 2021-11-20T05:32:17.476 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: :16.910525+0000 mon.smithi058 ( 2021-11-20T05:32:17.476 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: mon.0) 316 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8390f146-b7e1-4039-83d3-7ed1c832dc8e"}]': finished 2021-11-20T05:32:17.476 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: 2021-11-20T05:32:17.477 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: cluster 2021-11 2021-11-20T05:32:17.477 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: -20T05:32:16. 2021-11-20T05:32:17.477 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: 910579+0000 mon.smithi058 (mon.0) 317 : 2021-11-20T05:32:17.477 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: cluster [DBG] osdmap e19: 4 total, 3 up, 4 in 2021-11-20T05:32:17.477 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:32:17.478 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: 11-20T05:32: 2021-11-20T05:32:17.478 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: 16.910835+0000 mon.smithi058 (mon.0) 318 2021-11-20T05:32:17.478 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T05:32:17.478 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: audit 2021-11-20T05:32:17.479 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: 2021-11-20T05: 2021-11-20T05:32:17.479 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: 32:16.911109+0000 mon.smithi058 (mon. 2021-11-20T05:32:17.479 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: 0) 319 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:17.479 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: audit 2021-11-20T05:32:17.479 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: 2021-11-20T05: 2021-11-20T05:32:17.480 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: 32:17.057754+0000 mon.smithi058 (mon. 2021-11-20T05:32:17.481 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:17 smithi058 conmon[33537]: 0) 320 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:17.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:17 smithi187 conmon[36358]: cluster 2021-11-20T05:32:16.188824+0000 mgr.smithi058.zagdlk (mgr.14164) 114 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T05:32:17.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:17 smithi187 conmon[36358]: audit 2021-11-20T05:32:16.200230+0000 mon.smithi058 (mon.0) 310 : audit [INF] from='osd.2 [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi058", "root=default"]}]': finished 2021-11-20T05:32:17.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:17 smithi187 conmon[36358]: cluster 2021-11-20T05:32:16.200291+0000 mon.smithi058 (mon.0) 311 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2021-11-20T05:32:17.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:17 smithi187 conmon[36358]: audit 2021-11-20T05:32:16.200550+0000 mon.smithi058 (mon.0) 312 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T05:32:17.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:17 smithi187 conmon[36358]: audit 2021-11-20T05:32:16.203747+0000 mon.smithi058 (mon.0) 313 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T05:32:17.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:17 smithi187 conmon[36358]: audit 2021-11-20T05:32:16.905937+0000 mon.smithi187 (mon.1) 2 : audit [INF] from='client.? 172.21.15.187:0/3062992476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8390f146-b7e1-4039-83d3-7ed1c832dc8e"}]: dispatch 2021-11-20T05:32:17.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:17 smithi187 conmon[36358]: audit 2021-11-20T05:32:16.906472+0000 mon.smithi058 (mon.0) 314 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8390f146-b7e1-4039-83d3-7ed1c832dc8e"}]: dispatch 2021-11-20T05:32:17.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:17 smithi187 conmon[36358]: cluster 2021-11-20T05:32:16.910472+0000 mon.smithi058 (mon.0) 315 : cluster [INF] osd.2 [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076] boot 2021-11-20T05:32:17.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:17 smithi187 conmon[36358]: audit 2021-11-20T05:32:16.910525+0000 mon.smithi058 (mon.0) 316 : audit [INF] 2021-11-20T05:32:17.702 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:17 smithi187 conmon[36358]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8390f146-b7e1-4039-83d3-7ed1c832dc8e"}]': finished 2021-11-20T05:32:17.702 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:17 smithi187 conmon[36358]: cluster 2021-11-20T05:32:16.910579+0000 mon.smithi058 (mon.0) 317 : cluster [DBG] osdmap e19: 4 total, 3 up, 4 in 2021-11-20T05:32:17.703 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:17 smithi187 conmon[36358]: audit 2021-11-20T05:32:16.910835+0000 mon.smithi058 (mon.0) 318 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T05:32:17.703 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:17 smithi187 conmon[36358]: audit 2021-11-20T05:32:16.911109+0000 mon.smithi058 (mon.0) 319 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:17.704 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:17 smithi187 conmon[36358]: audit 2021-11-20T05:32:17.057754+0000 mon.smithi058 (mon.0) 320 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:18.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:18 smithi058 conmon[33537]: cluster 2021-11-20T05:32:15.227127+0000 osd.2 (osd.2 2021-11-20T05:32:18.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:18 smithi058 conmon[33537]: ) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T05:32:18.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:18 smithi058 conmon[33537]: cluster 2021-11-20T05:32:15.227269+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T05:32:18.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:18 smithi058 conmon[33537]: audit 2021-11-20T05:32:17.337217+0000 mon.smithi187 (mon.1 2021-11-20T05:32:18.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:18 smithi058 conmon[33537]: ) 3 : audit [DBG] from='client.? 172.21.15.187:0/3351019391' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T05:32:18.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:18 smithi187 conmon[36358]: cluster 2021-11-20T05:32:15.227127+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T05:32:18.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:18 smithi187 conmon[36358]: 2021-11-20T05:32:18.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:18 smithi187 conmon[36358]: cluster 2021-11-20T05:32:15.227269+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T05:32:18.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:18 smithi187 conmon[36358]: audit 2021-11-20T05:32:17.337217+0000 mon.smithi187 (mon.1) 3 : audit [DBG] from='client.? 172.21.15.187:0/3351019391' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T05:32:19.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:19 smithi058 conmon[33537]: cluster 2021-11-20T05:32:18.189339+0000 2021-11-20T05:32:19.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:19 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 115 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:19.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:19 smithi058 conmon[33537]: cluster 2021-11-20T05:32:18.205393+0000 mon.smithi058 (mon.0) 321 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2021-11-20T05:32:19.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:19 smithi058 conmon[33537]: audit 2021-11-20T05:32:18.205675+0000 mon.smithi058 (mon.0) 322 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:19.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:19 smithi187 conmon[36358]: cluster 2021-11-20T05:32:18.189339+0000 mgr.smithi058.zagdlk (mgr.14164) 115 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:19.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:19 smithi187 conmon[36358]: cluster 2021-11-20T05:32:18.205393+0000 mon.smithi058 (mon.0) 321 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2021-11-20T05:32:19.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:19 smithi187 conmon[36358]: audit 2021-11-20T05:32:18.205675+0000 mon.smithi058 (mon.0) 322 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:20.517 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:20 smithi058 conmon[33537]: audit 2021-11-20T05:32:19 2021-11-20T05:32:20.517 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:20 smithi058 conmon[33537]: .288534+0000 mon.smithi058 (mon.0) 323 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:20.568 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:20 smithi187 conmon[36358]: audit 2021-11-20T05:32:19.288534+0000 mon.smithi058 (mon.0) 323 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:21.544 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:21 smithi187 conmon[36358]: cluster 2021-11-20T05:32:20.189866+0000 mgr.smithi058.zagdlk 2021-11-20T05:32:21.544 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:21 smithi187 conmon[36358]: (mgr.14164) 116 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:21.619 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:21 smithi058 conmon[33537]: cluster 2021-11-20T05:32:20.189866+0000 mgr.smithi058.zagdlk (mgr.14164) 116 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:23.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:23 smithi187 conmon[36358]: cluster 2021-11-20T05:32:22.190139+0000 mgr.smithi058.zagdlk (mgr.14164) 117 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:23.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:23 smithi187 conmon[36358]: audit 2021-11-20T05:32:22.319098+0000 mon.smithi058 (mon.0) 324 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:23.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:23 smithi187 conmon[36358]: audit 2021-11-20T05:32:22.689205+0000 mon.smithi058 (mon.0) 325 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:23.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:23 smithi058 conmon[33537]: cluster 2021-11-20T05:32:22.190139+0000 mgr.smithi058.zagdlk (mgr.14164) 117 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:23.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:23 smithi058 conmon[33537]: audit 2021-11-20T05:32:22.319098+0000 mon.smithi058 (mon.0) 324 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:23.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:23 smithi058 conmon[33537]: audit 2021-11-20T05:32:22.689205+0000 mon.smithi058 (mon.0) 325 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:24.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:24 smithi187 conmon[36358]: audit 2021-11-20T05:32:23 2021-11-20T05:32:24.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:24 smithi187 conmon[36358]: .419883+0000 mon.smithi058 (mon.0) 326 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-11-20T05:32:24.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:24 smithi187 conmon[36358]: audit 2021-11-20T05:32:23.420759+0000 mon.smithi058 (mon.0) 327 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:32:24.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:24 smithi058 conmon[33537]: audit 2021-11-20T05:32:23.419883+0000 mon.smithi058 (mon.0) 326 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-11-20T05:32:24.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:24 smithi058 conmon[33537]: 2021-11-20T05:32:24.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:24 smithi058 conmon[33537]: audit 2021-11-20T05:32:23.420759+0000 mon.smithi058 (mon.0) 327 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:32:25.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:25 smithi187 conmon[36358]: cephadm 2021-11-20T05:32:23.421537+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:32:25.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:25 smithi187 conmon[36358]: mgr.14164) 118 : cephadm [INF] Deploying daemon osd.3 on smithi187 2021-11-20T05:32:25.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:25 smithi187 conmon[36358]: cluster 2021-11-20T05:32:24.190524+0000 mgr.smithi058.zagdlk (mgr.14164) 119 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:25.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:25 smithi058 conmon[33537]: cephadm 2021-11-20T05:32:23.421537+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:32:25.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:25 smithi058 conmon[33537]: mgr.14164) 118 : cephadm [INF] Deploying daemon osd.3 on smithi187 2021-11-20T05:32:25.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:25 smithi058 conmon[33537]: cluster 2021-11-20T05:32:24.190524+0000 2021-11-20T05:32:25.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:25 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 119 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:26.609 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 3 on host 'smithi187' 2021-11-20T05:32:26.611 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:26.607+0000 7f9bb17fa700 1 -- 172.21.15.187:0/163777562 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9b94000bc0 con 0x7f9ba405c2b0 2021-11-20T05:32:26.611 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:26.610+0000 7f9bbb220700 1 -- 172.21.15.187:0/163777562 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9ba405c2b0 msgr2=0x7f9ba405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:32:26.611 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:26.610+0000 7f9bbb220700 1 --2- 172.21.15.187:0/163777562 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9ba405c2b0 0x7f9ba405e770 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f9ba0009f50 tx=0x7f9ba0009b30).stop 2021-11-20T05:32:26.611 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:26.610+0000 7f9bbb220700 1 -- 172.21.15.187:0/163777562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bb406a350 msgr2=0x7f9bb4109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:32:26.612 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:26.610+0000 7f9bbb220700 1 --2- 172.21.15.187:0/163777562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bb406a350 0x7f9bb4109120 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f9ba80026d0 tx=0x7f9ba8009de0).stop 2021-11-20T05:32:26.612 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:26.610+0000 7f9bbb220700 1 -- 172.21.15.187:0/163777562 shutdown_connections 2021-11-20T05:32:26.612 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:26.610+0000 7f9bbb220700 1 --2- 172.21.15.187:0/163777562 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9ba405c2b0 0x7f9ba405e770 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:26.612 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:26.610+0000 7f9bbb220700 1 --2- 172.21.15.187:0/163777562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9bb406a350 0x7f9bb4109120 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:26.613 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:26.610+0000 7f9bbb220700 1 --2- 172.21.15.187:0/163777562 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9bb406b960 0x7f9bb4109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:26.613 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:26.610+0000 7f9bbb220700 1 -- 172.21.15.187:0/163777562 >> 172.21.15.187:0/163777562 conn(0x7f9bb40f1e40 msgr2=0x7f9bb40fce80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:32:26.613 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:26.610+0000 7f9bbb220700 1 -- 172.21.15.187:0/163777562 shutdown_connections 2021-11-20T05:32:26.613 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:26.610+0000 7f9bbb220700 1 -- 172.21.15.187:0/163777562 wait complete. 2021-11-20T05:32:27.051 DEBUG:teuthology.orchestra.run.smithi187:osd.3> sudo journalctl -f -n 0 -u ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.3.service 2021-11-20T05:32:27.054 INFO:tasks.cephadm:Deploying osd.4 on smithi187 with /dev/vg_nvme/lv_3... 2021-11-20T05:32:27.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-11-20T05:32:27.093 INFO:journalctl@ceph.osd.3.smithi187.stdout:-- Logs begin at Sat 2021-11-20 05:17:59 UTC. -- 2021-11-20T05:32:27.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:27 smithi187 conmon[36358]: cluster 2021-11-20T05:32:26.190931+0000 mgr.smithi058.zagdlk (mgr.14164) 120 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:27.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:27 smithi187 conmon[36358]: audit 2021-11-20T05:32:26.598269+0000 mon.smithi058 (mon.0) 328 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:32:27.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:27 smithi187 conmon[36358]: audit 2021-11-20T05:32:26.602606+0000 mon.smithi058 (mon.0) 329 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:27.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:27 smithi187 conmon[36358]: audit 2021-11-20T05:32:26.607161+0000 mon.smithi058 (mon.0) 330 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:27.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:27 smithi187 conmon[36358]: audit 2021-11-20T05:32:26.614301+0000 mon.smithi058 (mon.0) 331 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-11-20T05:32:27.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:27 smithi058 conmon[33537]: cluster 2021-11-20T05:32:26.190931+0000 mgr.smithi058.zagdlk (mgr.14164) 120 : cluster 2021-11-20T05:32:27.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:27 smithi058 conmon[33537]: [DBG] pgmap v82: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:27.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:27 smithi058 conmon[33537]: audit 2021-11-20T05:32:26.598269+0000 mon.smithi058 (mon.0) 328 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:32:27.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:27 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:32:27.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:27 smithi058 conmon[33537]: -20T05:32:26.602606+0000 mon.smithi058 (mon.0) 329 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:27.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:27 smithi058 conmon[33537]: audit 2021-11-20T05:32 2021-11-20T05:32:27.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:27 smithi058 conmon[33537]: :26.607161+0000 mon.smithi058 (mon.0) 330 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:27.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:27 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:32:27.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:27 smithi058 conmon[33537]: 11-20T05:32:26.614301+0000 mon.smithi058 (mon. 2021-11-20T05:32:27.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:27 smithi058 conmon[33537]: 0) 331 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-11-20T05:32:28.789 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:28 smithi187 conmon[36358]: audit 2021-11-20T05:32:27.609107+0000 2021-11-20T05:32:28.789 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:28 smithi187 conmon[36358]: mon.smithi058 (mon.0) 332 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-11-20T05:32:28.790 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:28 smithi187 conmon[36358]: cluster 2021-11-20T05:32:27.609157+0000 mon.smithi058 (mon.0) 333 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2021-11-20T05:32:28.790 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:28 smithi187 conmon[36358]: audit 2021-11-20T05:32:27.609275+0000 mon.smithi058 (mon.0) 334 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:28.791 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:28 smithi187 conmon[36358]: audit 2021-11-20T05:32:27.609982+0000 mon.smithi058 (mon.0) 335 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-11-20T05:32:28.791 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:28 smithi187 conmon[36358]: cluster 2021-11-20T05:32:28.191311+0000 mgr.smithi058.zagdlk (mgr.14164) 121 : cluster [DBG] pgmap v84: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:28.969 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:28 smithi058 conmon[33537]: audit 2021-11-20T05:32:27. 2021-11-20T05:32:28.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:28 smithi058 conmon[33537]: 609107+0000 mon.smithi058 (mon.0) 332 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-11-20T05:32:28.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:28 smithi058 conmon[33537]: cluster 2021-11-20T05:32:27.609157+0000 mon.smithi058 (mon.0) 333 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2021-11-20T05:32:28.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:28 smithi058 conmon[33537]: audit 2021-11-20T05:32:27.609275+0000 mon.smithi058 (mon.0) 334 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:28.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:28 smithi058 conmon[33537]: audit 2021-11-20T05:32:27.609982+0000 mon.smithi058 (mon.0) 335 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-11-20T05:32:28.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:28 smithi058 conmon[33537]: cluster 2021-11-20T05:32:28.191311+0000 mgr.smithi058.zagdlk (mgr.14164) 121 : cluster [DBG] pgmap v84: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:29.299 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-11-20T05:32:29.300 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-11-20T05:32:29.300 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2021-11-20T05:32:29.300 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2021-11-20T05:32:29.301 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131771 s, 796 MB/s 2021-11-20T05:32:29.301 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2021-11-20T05:32:29.936 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:29 smithi187 conmon[36358]: audit 2021-11-20T05:32:28.612858+0000 mon.smithi058 (mon.0) 336 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-11-20T05:32:29.936 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:29 smithi187 conmon[36358]: cluster 2021-11-20T05:32:28.612944+0000 mon.smithi058 (mon.0) 337 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-11-20T05:32:29.936 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:29 smithi187 conmon[36358]: audit 2021-11-20T05:32:28.613239+0000 mon.smithi058 (mon.0) 338 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:29.969 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:29 smithi058 conmon[33537]: audit 2021-11-20T05:32:28.612858+0000 2021-11-20T05:32:29.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:29 smithi058 conmon[33537]: mon.smithi058 (mon.0) 336 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-11-20T05:32:29.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:29 smithi058 conmon[33537]: cluster 2021-11- 2021-11-20T05:32:29.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:29 smithi058 conmon[33537]: 20T05:32:28.612944+0000 mon.smithi058 (mon.0) 337 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-11-20T05:32:29.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:29 smithi058 conmon[33537]: audit 2021-11-20 2021-11-20T05:32:29.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:29 smithi058 conmon[33537]: T05:32:28.613239+0000 mon.smithi058 (mon. 2021-11-20T05:32:29.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:29 smithi058 conmon[33537]: 0) 338 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:30.014 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi187:vg_nvme/lv_3 2021-11-20T05:32:30.736 INFO:journalctl@ceph.osd.3.smithi187.stdout:Nov 20 05:32:30 smithi187 conmon[42957]: debug 2021-11-20T05:32:30.512+0000 7f6859b6e080 -1 osd.3 0 log_to_monitors {default=true} 2021-11-20T05:32:31.365 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:31 smithi187 conmon[36358]: audit 2021-11-20T05:32:30.038655+0000 mon.smithi058 (mon.0) 339 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:31.365 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:31 smithi187 conmon[36358]: cluster 2021-11-20T05:32:30.191645+0000 mgr.smithi058.zagdlk (mgr.14164) 122 : cluster [DBG] pgmap v86: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:31.365 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:31 smithi187 conmon[36358]: cluster 2021-11-20T05:32:30.196681+0000 mon.smithi058 (mon.0) 340 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-11-20T05:32:31.366 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:31 smithi187 conmon[36358]: audit 2021-11-20T05:32:30.196808+0000 mon.smithi058 (mon.0) 341 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:31.366 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:31 smithi187 conmon[36358]: audit 2021-11-20T05:32:30.516885+0000 mon.smithi187 (mon.1) 4 : audit [INF] from='osd.3 [v2:172.21.15.187:6800/1941886381,v1:172.21.15.187:6801/1941886381]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-20T05:32:31.366 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:31 smithi187 conmon[36358]: audit 2021-11-20T05:32:30.517406+0000 mon.smithi058 (mon.0) 342 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-20T05:32:31.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:31 smithi058 conmon[33537]: audit 2021-11-20T05:32:30.038655+0000 mon.smithi058 (mon.0) 339 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:31.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:31 smithi058 conmon[33537]: cluster 2021-11-20T05:32:30.191645+0000 mgr.smithi058.zagdlk (mgr.14164) 122 : cluster [DBG] pgmap v86: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:31.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:31 smithi058 conmon[33537]: cluster 2021-11-20T05:32:30.196681+0000 mon.smithi058 (mon.0) 340 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-11-20T05:32:31.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:31 smithi058 conmon[33537]: audit 2021-11-20T05:32:30.196808+0000 mon.smithi058 (mon.0) 341 : audit [DBG] 2021-11-20T05:32:31.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:31 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:31.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:31 smithi058 conmon[33537]: audit 2021-11-20T05:32:30.516885+0000 mon.smithi187 (mon.1) 4 : audit [INF] from='osd.3 [v2:172.21.15.187:6800/1941886381,v1:172.21.15.187:6801/1941886381]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-20T05:32:31.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:31 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:32:31.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:31 smithi058 conmon[33537]: 11-20T05:32: 2021-11-20T05:32:31.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:31 smithi058 conmon[33537]: 30.517406+0000 mon.smithi058 (mon 2021-11-20T05:32:31.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:31 smithi058 conmon[33537]: .0) 342 : audit [INF] 2021-11-20T05:32:31.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:31 smithi058 conmon[33537]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-20T05:32:32.243 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:32 smithi187 conmon[36358]: audit 2021-11-20T05:32:31.197949+0000 mon.smithi058 (mon.0) 343 : audit 2021-11-20T05:32:32.243 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:32 smithi187 conmon[36358]: [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-11-20T05:32:32.243 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:32 smithi187 conmon[36358]: cluster 2021-11-20T05:32:31.198037+0000 mon.smithi058 (mon.0) 344 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2021-11-20T05:32:32.244 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:32 smithi187 conmon[36358]: audit 2021-11-20T05:32:31.198186+0000 mon.smithi058 (mon.0) 345 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:32.244 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:32 smithi187 conmon[36358]: audit 2021-11-20T05:32:31.198603+0000 mon.smithi187 (mon.1) 5 : audit [INF] from='osd.3 [v2:172.21.15.187:6800/1941886381,v1:172.21.15.187:6801/1941886381]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2021-11-20T05:32:32.244 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:32 smithi187 conmon[36358]: audit 2021-11-20T05:32:31.198935+0000 mon.smithi058 (mon.0) 346 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2021-11-20T05:32:32.244 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:32 smithi187 conmon[36358]: audit 2021-11-20T05:32:31.465753+0000 mon.smithi058 (mon.0) 347 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:32.245 INFO:journalctl@ceph.osd.3.smithi187.stdout:Nov 20 05:32:32 smithi187 conmon[42957]: debug 2021-11-20T05:32:32.204+0000 7f68500df700 -1 osd.3 0 waiting for initial osdmap 2021-11-20T05:32:32.245 INFO:journalctl@ceph.osd.3.smithi187.stdout:Nov 20 05:32:32 smithi187 conmon[42957]: debug 2021-11-20T05:32:32.210+0000 7f6848853700 -1 osd.3 25 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T05:32:32.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:32 smithi058 conmon[33537]: audit 2021-11-20T05:32:32.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:32 smithi058 conmon[33537]: 2021-11-20T05:32:31.197949+0000 mon.smithi058 (mon.0) 343 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-11-20T05:32:32.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:32 smithi058 conmon[33537]: cluster 2021-11-20T05:32:32.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:32 smithi058 conmon[33537]: 2021-11-20T05:32:31.198037+0000 2021-11-20T05:32:32.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:32 smithi058 conmon[33537]: mon.smithi058 (mon.0) 344 : cluster [DBG] 2021-11-20T05:32:32.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:32 smithi058 conmon[33537]: osdmap e24: 4 total, 3 up, 4 in 2021-11-20T05:32:32.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:32 smithi058 conmon[33537]: audit 2021-11-20T05:32:32.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:32 smithi058 conmon[33537]: 2021-11-20T05:32:31.198186+0000 mon.smithi058 (mon.0) 345 2021-11-20T05:32:32.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:32 smithi058 conmon[33537]: : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:32.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:32 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:32:32.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:32 smithi058 conmon[33537]: 11-20T05:32:31.198603+0000 mon.smithi187 (mon. 2021-11-20T05:32:32.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:32 smithi058 conmon[33537]: 1) 5 : audit [INF] from='osd.3 [v2:172.21.15.187:6800/1941886381,v1:172.21.15.187:6801/1941886381]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2021-11-20T05:32:32.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:32 smithi058 conmon[33537]: audit 2021 2021-11-20T05:32:32.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:32 smithi058 conmon[33537]: -11-20T05:32:31.198935+0000 mon.smithi058 (mon 2021-11-20T05:32:32.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:32 smithi058 conmon[33537]: .0) 346 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2021-11-20T05:32:32.475 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:32 smithi058 conmon[33537]: audit 2021-11-20T05:32:31.465753+0000 mon.smithi058 (mon.0) 347 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:32.550 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.543+0000 7f63b5798700 1 -- 172.21.15.187:0/494825451 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b00f66e0 msgr2=0x7f63b00f6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:32:32.550 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.543+0000 7f63b5798700 1 --2- 172.21.15.187:0/494825451 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b00f66e0 0x7f63b00f6b00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f63a00099e0 tx=0x7f63a0004d10).stop 2021-11-20T05:32:32.552 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.543+0000 7f63b5798700 1 -- 172.21.15.187:0/494825451 shutdown_connections 2021-11-20T05:32:32.553 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.543+0000 7f63b5798700 1 --2- 172.21.15.187:0/494825451 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63b00f7980 0x7f63b00f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:32.554 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.543+0000 7f63b5798700 1 --2- 172.21.15.187:0/494825451 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b00f66e0 0x7f63b00f6b00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:32.555 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.543+0000 7f63b5798700 1 -- 172.21.15.187:0/494825451 >> 172.21.15.187:0/494825451 conn(0x7f63b00f1e40 msgr2=0x7f63b00f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:32:32.555 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.543+0000 7f63b5798700 1 -- 172.21.15.187:0/494825451 shutdown_connections 2021-11-20T05:32:32.556 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.543+0000 7f63b5798700 1 -- 172.21.15.187:0/494825451 wait complete. 2021-11-20T05:32:32.556 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.544+0000 7f63b5798700 1 Processor -- start 2021-11-20T05:32:32.557 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.544+0000 7f63b5798700 1 -- start start 2021-11-20T05:32:32.557 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.544+0000 7f63b5798700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b00f66e0 0x7f63b01070f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:32:32.557 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.544+0000 7f63b5798700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63b00f7980 0x7f63b01075e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:32:32.558 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.544+0000 7f63b5798700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b0107b90 con 0x7f63b00f7980 2021-11-20T05:32:32.559 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.544+0000 7f63b5798700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b0107cd0 con 0x7f63b00f66e0 2021-11-20T05:32:32.559 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.544+0000 7f63aeffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b00f66e0 0x7f63b01070f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:32:32.559 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.544+0000 7f63aeffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b00f66e0 0x7f63b01070f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44382/0 (socket says 172.21.15.187:44382) 2021-11-20T05:32:32.560 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.544+0000 7f63aeffd700 1 -- 172.21.15.187:0/1240284143 learned_addr learned my addr 172.21.15.187:0/1240284143 (peer_addr_for_me v2:172.21.15.187:0/0) 2021-11-20T05:32:32.560 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.544+0000 7f63ae7fc700 1 --2- 172.21.15.187:0/1240284143 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63b00f7980 0x7f63b01075e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:32:32.560 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.544+0000 7f63aeffd700 1 -- 172.21.15.187:0/1240284143 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63b00f7980 msgr2=0x7f63b01075e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:32:32.560 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.544+0000 7f63aeffd700 1 --2- 172.21.15.187:0/1240284143 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63b00f7980 0x7f63b01075e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:32.561 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.544+0000 7f63aeffd700 1 -- 172.21.15.187:0/1240284143 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a0009640 con 0x7f63b00f66e0 2021-11-20T05:32:32.561 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.544+0000 7f63aeffd700 1 --2- 172.21.15.187:0/1240284143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b00f66e0 0x7f63b01070f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f63a00078b0 tx=0x7f63a0004d10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:32:32.561 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.545+0000 7f63977fe700 1 -- 172.21.15.187:0/1240284143 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63a000e030 con 0x7f63b00f66e0 2021-11-20T05:32:32.561 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.545+0000 7f63b5798700 1 -- 172.21.15.187:0/1240284143 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b0108610 con 0x7f63b00f66e0 2021-11-20T05:32:32.562 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.545+0000 7f63b5798700 1 -- 172.21.15.187:0/1240284143 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b0108af0 con 0x7f63b00f66e0 2021-11-20T05:32:32.562 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.545+0000 7f63977fe700 1 -- 172.21.15.187:0/1240284143 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63a0007d50 con 0x7f63b00f66e0 2021-11-20T05:32:32.562 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.545+0000 7f63977fe700 1 -- 172.21.15.187:0/1240284143 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63a0018070 con 0x7f63b00f66e0 2021-11-20T05:32:32.564 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.546+0000 7f63b5798700 1 -- 172.21.15.187:0/1240284143 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b0045ad0 con 0x7f63b00f66e0 2021-11-20T05:32:32.564 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.546+0000 7f63977fe700 1 -- 172.21.15.187:0/1240284143 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f63a0013790 con 0x7f63b00f66e0 2021-11-20T05:32:32.565 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.546+0000 7f63977fe700 1 --2- 172.21.15.187:0/1240284143 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f639805c2b0 0x7f639805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:32:32.565 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.546+0000 7f63977fe700 1 -- 172.21.15.187:0/1240284143 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 3408+0+0 (secure 0 0 0) 0x7f63a00773f0 con 0x7f63b00f66e0 2021-11-20T05:32:32.565 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.546+0000 7f63ae7fc700 1 --2- 172.21.15.187:0/1240284143 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f639805c2b0 0x7f639805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:32:32.565 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.547+0000 7f63ae7fc700 1 --2- 172.21.15.187:0/1240284143 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f639805c2b0 0x7f639805e770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f63a4004c40 tx=0x7f63a400b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:32:32.566 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.550+0000 7f63977fe700 1 -- 172.21.15.187:0/1240284143 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63a004a8f0 con 0x7f63b00f66e0 2021-11-20T05:32:32.736 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:32.733+0000 7f63b5798700 1 -- 172.21.15.187:0/1240284143 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f63b00fef80 con 0x7f639805c2b0 2021-11-20T05:32:33.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: cluster 2021-11-20T05:32:32.191968+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:32:33.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: mgr.14164) 123 : cluster [DBG] pgmap v89: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:33.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: audit 2021-11-20T05:32:32.199549+0000 mon.smithi058 (mon.0) 348 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2021-11-20T05:32:33.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: cluster 2021-11-20T05:32:32 2021-11-20T05:32:33.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: .199702+0000 mon.smithi058 (mon.0) 349 : cluster 2021-11-20T05:32:33.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: [DBG] osdmap e25: 4 total, 3 up, 4 in 2021-11-20T05:32:33.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: audit 2021-11-20T05 2021-11-20T05:32:33.475 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: :32:32.200185+0000 mon.smithi058 (mon 2021-11-20T05:32:33.475 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: .0) 350 : audit [DBG] 2021-11-20T05:32:33.476 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:33.476 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: audit 2021-11-20T05:32:33.477 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: 2021-11-20T 2021-11-20T05:32:33.477 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: 05:32:32.205947+0000 mon.smithi058 (mon.0) 351 2021-11-20T05:32:33.479 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:33.479 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: audit 2021-11-20T05:32: 2021-11-20T05:32:33.479 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: 32.736963+0000 mon.smithi058 (mon.0) 352 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T05:32:33.479 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: audit 2021-11-20T05:32:32.738825+0000 mon.smithi058 (mon.0) 353 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T05:32:33.480 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:33 smithi058 conmon[33537]: audit 2021-11-20T05:32:32.739690+0000 mon.smithi058 (mon.0) 354 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:32:33.496 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:33 smithi187 conmon[36358]: cluster 2021-11-20T05:32:32.191968+0000 mgr.smithi058.zagdlk (mgr.14164) 123 : cluster [DBG] pgmap v89: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:33.497 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:33 smithi187 conmon[36358]: audit 2021-11-20T05:32:32.199549+0000 mon.smithi058 (mon.0) 348 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2021-11-20T05:32:33.497 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:33 smithi187 conmon[36358]: cluster 2021-11-20T05:32:32.199702+0000 mon.smithi058 (mon.0) 349 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2021-11-20T05:32:33.498 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:33 smithi187 conmon[36358]: audit 2021-11-20T05:32:32.200185+0000 mon.smithi058 (mon.0) 350 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:33.498 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:33 smithi187 conmon[36358]: audit 2021-11-20T05:32:32.205947+0000 mon.smithi058 (mon.0) 351 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:33.498 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:33 smithi187 conmon[36358]: audit 2021-11-20T05:32:32.736963+0000 mon.smithi058 (mon.0) 352 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T05:32:33.498 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:33 smithi187 conmon[36358]: audit 2021-11-20T05:32:32.738825+0000 mon.smithi058 (mon.0) 353 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T05:32:33.499 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:33 smithi187 conmon[36358]: audit 2021-11-20T05:32:32.739690+0000 mon.smithi058 (mon.0) 354 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:32:34.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: cluster 2021-11-20T05:32:31.493668+0000 osd.3 (osd.3) 2021-11-20T05:32:34.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T05:32:34.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: cluster 2021-11-20T05:32:31.493766+0000 osd.3 (osd.3) 2 : cluster [DBG] 2021-11-20T05:32:34.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: purged_snaps scrub ok 2021-11-20T05:32:34.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: audit 2021-11-20T05:32:34.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: 2021-11-20T05:32:32.736031+0000 mgr.smithi058.zagdlk 2021-11-20T05:32:34.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: (mgr.14164) 124 : audit [DBG] from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:32:34.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: 2021-11-20T05:32:34.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: cephadm 2021 2021-11-20T05:32:34.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: -11-20T05:32:32.738497+0000 mgr.smithi058.zagdlk 2021-11-20T05:32:34.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: (mgr.14164) 125 : cephadm [INF] Applying service osd.None on host smithi187... 2021-11-20T05:32:34.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: 2021-11-20T05:32:34.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: cluster 2021-11-20T05:32:34.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: 2021-11-20T05:32:33.206080+0000 2021-11-20T05:32:34.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: mon.smithi058 (mon.0) 355 : 2021-11-20T05:32:34.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: cluster [INF] osd.3 [v2:172.21.15.187:6800/1941886381,v1:172.21.15.187:6801/1941886381] boot 2021-11-20T05:32:34.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: cluster 2021-11 2021-11-20T05:32:34.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: -20T05:32:33.206121+0000 2021-11-20T05:32:34.475 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: mon.smithi058 (mon.0) 356 : cluster [DBG] 2021-11-20T05:32:34.475 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: osdmap e26: 4 total, 4 up, 4 in 2021-11-20T05:32:34.475 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:32:34.475 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: -20T05:32:33.206589+0000 2021-11-20T05:32:34.476 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: mon.smithi058 (mon.0) 357 : audit [DBG] 2021-11-20T05:32:34.476 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:34.476 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: audit 2021-11-20T05: 2021-11-20T05:32:34.476 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: 32:33.618410+0000 mon.smithi058 (mon.0 2021-11-20T05:32:34.476 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: ) 358 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:34.477 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: audit 2021-11- 2021-11-20T05:32:34.477 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: 20T05:32:33.622277+0000 mon.smithi058 2021-11-20T05:32:34.477 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: (mon.0) 359 : audit [INF] 2021-11-20T05:32:34.477 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:34.477 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: audit 2021-11-20T05:32:34.478 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: 2021-11-20T05:32:33.626792+0000 2021-11-20T05:32:34.478 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: mon.smithi058 (mon.0) 360 : audit [DBG] 2021-11-20T05:32:34.478 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:32:34.478 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: cluster 2021-11-20T05:32:34.479 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: 2021-11-20T05:32:34.196361 2021-11-20T05:32:34.479 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 361 : cluster 2021-11-20T05:32:34.479 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:34 smithi058 conmon[33537]: [DBG] osdmap e27: 4 total, 4 up, 4 in 2021-11-20T05:32:34.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: cluster 2021-11-20T05:32:31.493668+0000 osd.3 (osd 2021-11-20T05:32:34.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: .3) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T05:32:34.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: cluster 2021-11-20T05:32:31.493766+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T05:32:34.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: audit 2021-11-20T05:32:32.736031+0000 mgr.smithi058.zagdlk (mgr.14164) 124 : audit [DBG] from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:32:34.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: cephadm 2021-11-20T05:32:32.738497+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:32:34.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: ) 125 : cephadm [INF] Applying service osd.None on host smithi187... 2021-11-20T05:32:34.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: cluster 2021-11-20 2021-11-20T05:32:34.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: T05:32:33.206080+0000 mon.smithi058 (mon. 2021-11-20T05:32:34.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: 0) 355 : cluster [INF] osd.3 [v2:172.21.15.187:6800/1941886381,v1:172.21.15.187:6801/1941886381] boot 2021-11-20T05:32:34.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: cluster 2021-11-20 2021-11-20T05:32:34.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: T05:32:33.206121+0000 mon.smithi058 (mon 2021-11-20T05:32:34.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: .0) 356 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2021-11-20T05:32:34.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: audit 2021-11- 2021-11-20T05:32:34.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: 20T05:32:33.206589+0000 mon.smithi058 ( 2021-11-20T05:32:34.702 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: mon.0) 357 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T05:32:34.702 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: 2021-11-20T05:32:34.702 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: audit 2021-11-20T05: 2021-11-20T05:32:34.702 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: 32:33.618410+0000 mon.smithi058 (mon. 2021-11-20T05:32:34.702 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: 0) 358 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:34.703 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: audit 2021-11 2021-11-20T05:32:34.703 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: -20T05:32:33.622277+0000 2021-11-20T05:32:34.703 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: mon.smithi058 (mon.0) 359 : audit 2021-11-20T05:32:34.703 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:34.704 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: audit 2021 2021-11-20T05:32:34.704 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: -11-20T05:32:33. 2021-11-20T05:32:34.704 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: 626792+0000 mon.smithi058 (mon.0) 360 2021-11-20T05:32:34.704 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:32:34.704 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: cluster 2021-11-20T05:32:34.705 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: 2021-11-20T05:32:34 2021-11-20T05:32:34.705 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: .196361+0000 mon.smithi058 (mon.0) 361 2021-11-20T05:32:34.705 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:34 smithi187 conmon[36358]: : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2021-11-20T05:32:35.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: cephadm 2021-11-20T05:32:33.613473 2021-11-20T05:32:35.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: +0000 mgr.smithi058.zagdlk (mgr.14164) 126 : cephadm [INF] Detected new or changed devices on smithi187 2021-11-20T05:32:35.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: cluster 2021-11-20T05:32:34.192293+0000 mgr.smithi058.zagdlk (mgr 2021-11-20T05:32:35.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: .14164) 127 : cluster [DBG] pgmap v92: 1 pgs: 1 remapped+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:35.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: audit 2021-11-20T05:32: 2021-11-20T05:32:35.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: 35.125120+0000 mon.smithi187 (mon.1) 6 : audit 2021-11-20T05:32:35.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: [INF] from='client.? 172.21.15.187:0/942584307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89dc0c2f-c52c-48ca-b3ab-792e299d494d"}]: dispatch 2021-11-20T05:32:35.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: audit 2021-11-20T05:32:35. 2021-11-20T05:32:35.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: 125598+0000 mon.smithi058 (mon.0) 362 : audit 2021-11-20T05:32:35.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89dc0c2f-c52c-48ca-b3ab-792e299d494d"}]: dispatch 2021-11-20T05:32:35.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: audit 2021-11-20T05:32:35.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: 2021-11-20T05:32:35.129233+0000 mon.smithi058 2021-11-20T05:32:35.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: (mon.0) 363 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89dc0c2f-c52c-48ca-b3ab-792e299d494d"}]': finished 2021-11-20T05:32:35.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: cluster 2021-11-20T 2021-11-20T05:32:35.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: 05:32:35.129297+0000 mon.smithi058 (mon.0) 364 : cluster [DBG] 2021-11-20T05:32:35.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: osdmap e28: 5 total, 4 up, 5 in 2021-11-20T05:32:35.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: audit 2021-11- 2021-11-20T05:32:35.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:35 smithi058 conmon[33537]: 20T05:32:35.129395+0000 mon.smithi058 (mon.0) 365 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T05:32:35.535 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:35 smithi187 conmon[36358]: cephadm 2021-11-20T05:32:33.613473+0000 mgr.smithi058.zagdlk (mgr.14164) 126 : cephadm [INF] Detected new or changed devices on smithi187 2021-11-20T05:32:35.535 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:35 smithi187 conmon[36358]: cluster 2021-11-20T05:32:34.192293+0000 mgr.smithi058.zagdlk (mgr.14164) 127 : cluster [DBG] pgmap v92: 1 pgs: 1 remapped+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T05:32:35.535 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:35 smithi187 conmon[36358]: audit 2021-11-20T05:32:35.125120+0000 mon.smithi187 (mon.1) 6 : audit [INF] from='client.? 172.21.15.187:0/942584307' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89dc0c2f-c52c-48ca-b3ab-792e299d494d"}]: dispatch 2021-11-20T05:32:35.536 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:35 smithi187 conmon[36358]: audit 2021-11-20T05:32:35.125598+0000 mon.smithi058 (mon.0) 362 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89dc0c2f-c52c-48ca-b3ab-792e299d494d"}]: dispatch 2021-11-20T05:32:35.536 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:35 smithi187 conmon[36358]: audit 2021-11-20T05:32:35.129233+0000 mon.smithi058 (mon.0) 363 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89dc0c2f-c52c-48ca-b3ab-792e299d494d"}]': finished 2021-11-20T05:32:35.536 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:35 smithi187 conmon[36358]: cluster 2021-11-20T05:32:35.129297+0000 mon.smithi058 (mon.0) 364 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2021-11-20T05:32:35.536 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:35 smithi187 conmon[36358]: audit 2021-11-20T05:32:35.129395+0000 mon.smithi058 (mon.0) 365 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T05:32:36.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:36 smithi058 conmon[33537]: audit 2021-11-20T05:32:35.535524+0000 mon.smithi187 (mon 2021-11-20T05:32:36.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:36 smithi058 conmon[33537]: .1) 7 : audit [DBG] from='client.? 172.21.15.187:0/2115532538' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T05:32:36.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:36 smithi187 conmon[36358]: audit 2021-11-20T05:32:35.535524+0000 mon.smithi187 (mon.1) 7 2021-11-20T05:32:36.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:36 smithi187 conmon[36358]: : audit [DBG] from='client.? 172.21.15.187:0/2115532538' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T05:32:37.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:37 smithi058 conmon[33537]: cluster 2021-11-20T05:32:36.192573+0000 2021-11-20T05:32:37.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:37 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 128 : cluster [DBG] pgmap v95: 1 pgs: 1 remapped+peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:37.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:37 smithi187 conmon[36358]: cluster 2021-11-20T05:32:36.192573+0000 mgr.smithi058.zagdlk (mgr.14164) 128 : cluster [DBG] pgmap v95: 1 pgs: 1 remapped+peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:39.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:39 smithi058 conmon[33537]: cluster 2021-11-20T05:32:38.192954+0000 mgr.smithi058.zagdlk (mgr.14164) 129 : cluster [DBG] pgmap v96: 1 pgs: 1 remapped+peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:39.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:39 smithi058 conmon[33537]: audit 2021-11-20T05:32:38.571156+0000 mon.smithi058 (mon.0) 366 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:39.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:39 smithi058 conmon[33537]: audit 2021-11-20T05:32:38.574180+0000 mon.smithi058 (mon.0) 367 : audit 2021-11-20T05:32:39.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:39 smithi058 conmon[33537]: [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:39.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:39 smithi187 conmon[36358]: cluster 2021-11-20T05:32:38.192954+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:32:39.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:39 smithi187 conmon[36358]: 129 : cluster [DBG] pgmap v96: 1 pgs: 1 remapped+peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:39.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:39 smithi187 conmon[36358]: audit 2021-11-20T05:32:38.571156+0000 mon.smithi058 (mon.0) 366 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:39.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:39 smithi187 conmon[36358]: audit 2021-11-20T05:32:38.574180+0000 mon.smithi058 (mon.0) 367 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:41.679 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:41 smithi187 conmon[36358]: cluster 2021-11-20T05:32:40.193343+0000 mgr.smithi058.zagdlk (mgr.14164) 130 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:41.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:41 smithi058 conmon[33537]: cluster 2021-11-20T05:32:40.193343+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:32:41.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:41 smithi058 conmon[33537]: 14164) 130 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:42.480 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:42 smithi187 conmon[36358]: audit 2021-11-20T05:32:41.747699+0000 mon.smithi058 (mon 2021-11-20T05:32:42.480 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:42 smithi187 conmon[36358]: .0) 368 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-11-20T05:32:42.480 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:42 smithi187 conmon[36358]: audit 2021-11-20T05:32:41.748612+0000 mon.smithi058 (mon.0) 369 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:32:42.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:42 smithi058 conmon[33537]: audit 2021-11-20T05:32:41.747699+0000 mon.smithi058 (mon.0) 368 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-11-20T05:32:42.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:42 smithi058 conmon[33537]: audit 2021-11-20T05:32:41.748612+0000 mon.smithi058 (mon.0) 369 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:32:43.579 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:43 smithi187 conmon[36358]: cephadm 2021-11-20T05:32:41.749364+0000 mgr.smithi058.zagdlk (mgr.14164) 131 : cephadm [INF] Deploying daemon osd.4 on smithi187 2021-11-20T05:32:43.580 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:43 smithi187 conmon[36358]: cluster 2021-11-20T05:32:42.193711+0000 mgr.smithi058.zagdlk (mgr.14164) 132 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:43.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:43 smithi058 conmon[33537]: cephadm 2021-11-20T05:32:41.749364+0000 mgr.smithi058.zagdlk (mgr.14164) 131 : cephadm [INF] Deploying daemon osd.4 on smithi187 2021-11-20T05:32:43.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:43 smithi058 conmon[33537]: cluster 2021-11-20T05:32:42.193711+0000 mgr.smithi058.zagdlk (mgr.14164) 132 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:45.143 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 4 on host 'smithi187' 2021-11-20T05:32:45.144 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:45.142+0000 7f63977fe700 1 -- 172.21.15.187:0/1240284143 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f63b00fef80 con 0x7f639805c2b0 2021-11-20T05:32:45.144 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:45.144+0000 7f63b5798700 1 -- 172.21.15.187:0/1240284143 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f639805c2b0 msgr2=0x7f639805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:32:45.145 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:45.144+0000 7f63b5798700 1 --2- 172.21.15.187:0/1240284143 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f639805c2b0 0x7f639805e770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f63a4004c40 tx=0x7f63a400b040).stop 2021-11-20T05:32:45.145 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:45.144+0000 7f63b5798700 1 -- 172.21.15.187:0/1240284143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b00f66e0 msgr2=0x7f63b01070f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:32:45.145 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:45.144+0000 7f63b5798700 1 --2- 172.21.15.187:0/1240284143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b00f66e0 0x7f63b01070f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f63a00078b0 tx=0x7f63a0004d10).stop 2021-11-20T05:32:45.145 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:45.144+0000 7f63b5798700 1 -- 172.21.15.187:0/1240284143 shutdown_connections 2021-11-20T05:32:45.145 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:45.144+0000 7f63b5798700 1 --2- 172.21.15.187:0/1240284143 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f639805c2b0 0x7f639805e770 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:45.146 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:45.144+0000 7f63b5798700 1 --2- 172.21.15.187:0/1240284143 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63b00f66e0 0x7f63b01070f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:45.146 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:45.144+0000 7f63b5798700 1 --2- 172.21.15.187:0/1240284143 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63b00f7980 0x7f63b01075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:45.146 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:45.144+0000 7f63b5798700 1 -- 172.21.15.187:0/1240284143 >> 172.21.15.187:0/1240284143 conn(0x7f63b00f1e40 msgr2=0x7f63b00fb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:32:45.146 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:45.145+0000 7f63b5798700 1 -- 172.21.15.187:0/1240284143 shutdown_connections 2021-11-20T05:32:45.147 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:45.145+0000 7f63b5798700 1 -- 172.21.15.187:0/1240284143 wait complete. 2021-11-20T05:32:45.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:45 smithi187 conmon[36358]: cluster 2021-11-20T05:32:44.194080+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:32:45.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:45 smithi187 conmon[36358]: 14164) 133 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:45.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:45 smithi187 conmon[36358]: audit 2021-11-20T05:32:45.131907+0000 mon.smithi058 (mon.0) 370 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:32:45.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:45 smithi187 conmon[36358]: audit 2021-11-20T05:32:45.136202+0000 mon.smithi058 (mon.0) 371 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:45.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:45 smithi187 conmon[36358]: audit 2021-11-20T05:32:45.140974+0000 mon.smithi058 (mon.0) 372 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:45.578 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:45 smithi058 conmon[33537]: cluster 2021-11-20T05:32:44.194080+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:32:45.579 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:45 smithi058 conmon[33537]: mgr.14164) 133 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:45.579 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:45 smithi058 conmon[33537]: audit 2021-11-20T05:32:45.131907+0000 mon.smithi058 (mon.0) 370 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:32:45.580 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:45 smithi058 conmon[33537]: audit 2021-11-20T05:32:45.136202+0000 mon.smithi058 (mon.0) 2021-11-20T05:32:45.580 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:45 smithi058 conmon[33537]: 371 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:45.580 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:45 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:32:45.581 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:45 smithi058 conmon[33537]: 11-20T05:32:45.140974+0000 mon.smithi058 2021-11-20T05:32:45.581 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:45 smithi058 conmon[33537]: (mon.0) 372 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:45.581 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:45 smithi058 conmon[33537]: 2021-11-20T05:32:45.591 DEBUG:teuthology.orchestra.run.smithi187:osd.4> sudo journalctl -f -n 0 -u ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.4.service 2021-11-20T05:32:45.594 INFO:tasks.cephadm:Deploying osd.5 on smithi187 with /dev/vg_nvme/lv_2... 2021-11-20T05:32:45.595 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-11-20T05:32:45.622 INFO:journalctl@ceph.osd.4.smithi187.stdout:-- Logs begin at Sat 2021-11-20 05:17:59 UTC. -- 2021-11-20T05:32:47.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:47 smithi187 conmon[36358]: cluster 2021-11-20T05:32:46.194527+0000 mgr.smithi058.zagdlk (mgr.14164) 134 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:47.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:47 smithi058 conmon[33537]: cluster 2021-11-20T05:32:46.194527+0000 2021-11-20T05:32:47.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:47 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 134 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:47.954 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-11-20T05:32:47.954 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-11-20T05:32:47.954 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2021-11-20T05:32:47.955 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2021-11-20T05:32:47.955 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122942 s, 853 MB/s 2021-11-20T05:32:47.955 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2021-11-20T05:32:48.445 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi187:vg_nvme/lv_2 2021-11-20T05:32:49.194 INFO:journalctl@ceph.osd.4.smithi187.stdout:Nov 20 05:32:49 smithi187 conmon[47832]: debug 2021-11-20T05:32:49.040+0000 7f858c315080 -1 osd.4 0 log_to_monitors {default=true} 2021-11-20T05:32:49.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:49 smithi187 conmon[36358]: cluster 2021-11-20T05:32:48.194865+0000 2021-11-20T05:32:49.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:49 smithi187 conmon[36358]: mgr.smithi058.zagdlk (mgr.14164) 135 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:49.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:49 smithi187 conmon[36358]: audit 2021-11-20T05:32:48.556081+0000 mon.smithi058 (mon.0) 373 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:49.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:49 smithi187 conmon[36358]: audit 2021-11-20T05:32:49.043546+0000 2021-11-20T05:32:49.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:49 smithi187 conmon[36358]: mon.smithi187 (mon.1) 8 : audit [INF] from='osd.4 [v2:172.21.15.187:6808/3483552299,v1:172.21.15.187:6809/3483552299]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-20T05:32:49.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:49 smithi187 conmon[36358]: audit 2021-11-20T05:32:49.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:49 smithi187 conmon[36358]: 2021-11-20T05:32:49.044155+0000 mon.smithi058 2021-11-20T05:32:49.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:49 smithi187 conmon[36358]: (mon.0) 374 : audit [INF] 2021-11-20T05:32:49.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:49 smithi187 conmon[36358]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-20T05:32:49.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:49 smithi058 conmon[33537]: cluster 2021-11-20T05:32:48.194865+0000 mgr.smithi058.zagdlk (mgr.14164) 135 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:49.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:49 smithi058 conmon[33537]: audit 2021-11-20T05:32:48.556081+0000 mon.smithi058 (mon.0) 373 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:49.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:49 smithi058 conmon[33537]: audit 2021-11-20T05:32:49.043546+0000 mon.smithi187 (mon.1) 8 : audit [INF] from='osd.4 [v2:172.21.15.187:6808/3483552299,v1:172.21.15.187:6809/3483552299]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-20T05:32:49.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:49 smithi058 conmon[33537]: audit 2021-11-20T05:32:49.044155+0000 mon.smithi058 (mon.0) 374 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-20T05:32:50.562 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:50 smithi187 conmon[36358]: audit 2021-11-20T05:32:49.557572+0000 mon.smithi058 (mon.0) 375 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-11-20T05:32:50.563 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:50 smithi187 conmon[36358]: cluster 2021-11-20T05:32:49.557639+0000 mon.smithi058 (mon.0) 376 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-11-20T05:32:50.564 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:50 smithi187 conmon[36358]: audit 2021-11-20T05:32:49.557756+0000 mon.smithi058 (mon.0) 377 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T05:32:50.564 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:50 smithi187 conmon[36358]: audit 2021-11-20T05:32:49.558169+0000 mon.smithi187 (mon.1) 9 : audit [INF] from='osd.4 [v2:172.21.15.187:6808/3483552299,v1:172.21.15.187:6809/3483552299]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2021-11-20T05:32:50.564 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:50 smithi187 conmon[36358]: audit 2021-11-20T05:32:49.558398+0000 mon.smithi058 (mon.0) 378 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2021-11-20T05:32:50.565 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:50 smithi187 conmon[36358]: audit 2021-11-20T05:32:49.899005+0000 mon.smithi058 (mon.0) 379 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:50.565 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:50 smithi187 conmon[36358]: cluster 2021-11-20T05:32:50.195191+0000 mgr.smithi058.zagdlk (mgr 2021-11-20T05:32:50.565 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:50 smithi187 conmon[36358]: .14164) 136 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:50.879 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.879+0000 7f31c5a7f700 1 -- 172.21.15.187:0/1376337020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31c00f7980 msgr2=0x7f31c00f7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:32:50.879 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.879+0000 7f31c5a7f700 1 --2- 172.21.15.187:0/1376337020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31c00f7980 0x7f31c00f7de0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f31b0004660 tx=0x7f31b0009b30).stop 2021-11-20T05:32:50.880 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.879+0000 7f31c5a7f700 1 -- 172.21.15.187:0/1376337020 shutdown_connections 2021-11-20T05:32:50.880 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.879+0000 7f31c5a7f700 1 --2- 172.21.15.187:0/1376337020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31c00f7980 0x7f31c00f7de0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:50.880 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.879+0000 7f31c5a7f700 1 --2- 172.21.15.187:0/1376337020 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f31c00f66e0 0x7f31c00f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:50.880 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.879+0000 7f31c5a7f700 1 -- 172.21.15.187:0/1376337020 >> 172.21.15.187:0/1376337020 conn(0x7f31c00f1e40 msgr2=0x7f31c00f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:32:50.881 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.879+0000 7f31c5a7f700 1 -- 172.21.15.187:0/1376337020 shutdown_connections 2021-11-20T05:32:50.881 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.879+0000 7f31c5a7f700 1 -- 172.21.15.187:0/1376337020 wait complete. 2021-11-20T05:32:50.881 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.879+0000 7f31c5a7f700 1 Processor -- start 2021-11-20T05:32:50.882 INFO:journalctl@ceph.osd.4.smithi187.stdout:Nov 20 05:32:50 smithi187 conmon[47832]: debug 2021-11-20T05:32:50.564+0000 7f8582886700 -1 osd.4 0 waiting for initial osdmap 2021-11-20T05:32:50.882 INFO:journalctl@ceph.osd.4.smithi187.stdout:Nov 20 05:32:50 smithi187 conmon[47832]: debug 2021-11-20T05:32:50.571+0000 7f857f002700 -1 osd.4 30 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T05:32:50.886 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31c5a7f700 1 -- start start 2021-11-20T05:32:50.887 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31c5a7f700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f31c00f66e0 0x7f31c0106fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:32:50.887 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31c5a7f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31c00f7980 0x7f31c0107490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:32:50.887 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31c5a7f700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c0107a40 con 0x7f31c00f66e0 2021-11-20T05:32:50.888 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31c5a7f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c0107b80 con 0x7f31c00f7980 2021-11-20T05:32:50.888 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31beffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31c00f7980 0x7f31c0107490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:32:50.889 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31beffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31c00f7980 0x7f31c0107490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44448/0 (socket says 172.21.15.187:44448) 2021-11-20T05:32:50.889 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31beffd700 1 -- 172.21.15.187:0/1863108242 learned_addr learned my addr 172.21.15.187:0/1863108242 (peer_addr_for_me v2:172.21.15.187:0/0) 2021-11-20T05:32:50.889 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31bf7fe700 1 --2- 172.21.15.187:0/1863108242 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f31c00f66e0 0x7f31c0106fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:32:50.890 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31beffd700 1 -- 172.21.15.187:0/1863108242 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f31c00f66e0 msgr2=0x7f31c0106fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:32:50.890 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31beffd700 1 --2- 172.21.15.187:0/1863108242 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f31c00f66e0 0x7f31c0106fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:32:50.890 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31beffd700 1 -- 172.21.15.187:0/1863108242 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b0005040 con 0x7f31c00f7980 2021-11-20T05:32:50.890 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31beffd700 1 --2- 172.21.15.187:0/1863108242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31c00f7980 0x7f31c0107490 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f31b0009f50 tx=0x7f31b0009b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:32:50.891 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31a7fff700 1 -- 172.21.15.187:0/1863108242 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31b0018070 con 0x7f31c00f7980 2021-11-20T05:32:50.891 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31c5a7f700 1 -- 172.21.15.187:0/1863108242 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c01084c0 con 0x7f31c00f7980 2021-11-20T05:32:50.891 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31c5a7f700 1 -- 172.21.15.187:0/1863108242 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c0108a20 con 0x7f31c00f7980 2021-11-20T05:32:50.891 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31a7fff700 1 -- 172.21.15.187:0/1863108242 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31b0013410 con 0x7f31c00f7980 2021-11-20T05:32:50.892 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.880+0000 7f31a7fff700 1 -- 172.21.15.187:0/1863108242 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31b001c3c0 con 0x7f31c00f7980 2021-11-20T05:32:50.892 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.881+0000 7f31c5a7f700 1 -- 172.21.15.187:0/1863108242 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c0045ad0 con 0x7f31c00f7980 2021-11-20T05:32:50.892 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.882+0000 7f31a7fff700 1 -- 172.21.15.187:0/1863108242 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f31b001d420 con 0x7f31c00f7980 2021-11-20T05:32:50.892 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.882+0000 7f31a7fff700 1 --2- 172.21.15.187:0/1863108242 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f31a805c300 0x7f31a805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:32:50.893 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.882+0000 7f31a7fff700 1 -- 172.21.15.187:0/1863108242 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3840+0+0 (secure 0 0 0) 0x7f31b0077670 con 0x7f31c00f7980 2021-11-20T05:32:50.893 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.882+0000 7f31bf7fe700 1 --2- 172.21.15.187:0/1863108242 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f31a805c300 0x7f31a805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:32:50.894 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.883+0000 7f31bf7fe700 1 --2- 172.21.15.187:0/1863108242 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f31a805c300 0x7f31a805e7c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f31c00f77e0 tx=0x7f31b4009c00).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:32:50.894 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:50.886+0000 7f31a7fff700 1 -- 172.21.15.187:0/1863108242 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31b004a8a0 con 0x7f31c00f7980 2021-11-20T05:32:50.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:50 smithi058 conmon[33537]: audit 2021-11-20T05:32:49.557572+0000 mon.smithi058 (mon.0) 375 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-11-20T05:32:50.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:50 smithi058 conmon[33537]: cluster 2021-11-20T05:32:49.557639+0000 mon.smithi058 (mon.0) 376 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-11-20T05:32:50.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:50 smithi058 conmon[33537]: audit 2021-11-20T05:32:49.557756+0000 mon.smithi058 (mon.0) 377 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T05:32:50.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:50 smithi058 conmon[33537]: audit 2021-11-20T05:32:49.558169+0000 mon.smithi187 (mon.1) 9 : audit [INF] from='osd.4 [v2:172.21.15.187:6808/3483552299,v1:172.21.15.187:6809/3483552299]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2021-11-20T05:32:50.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:50 smithi058 conmon[33537]: audit 2021-11-20T05:32:49.558398+0000 mon.smithi058 (mon.0) 378 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2021-11-20T05:32:50.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:50 smithi058 conmon[33537]: audit 2021-11-20T05:32:49.899005+0000 mon.smithi058 (mon.0) 379 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:50.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:50 smithi058 conmon[33537]: cluster 2021-11-20T05:32:50.195191+0000 mgr.smithi058.zagdlk (mgr.14164) 136 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:51.071 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:32:51.071+0000 7f31c5a7f700 1 -- 172.21.15.187:0/1863108242 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f31c00fef80 con 0x7f31a805c300 2021-11-20T05:32:51.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:51 smithi187 conmon[36358]: audit 2021-11-20T05:32:50.559727+0000 mon.smithi058 (mon 2021-11-20T05:32:51.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:51 smithi187 conmon[36358]: .0) 380 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2021-11-20T05:32:51.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:51 smithi187 conmon[36358]: cluster 2021-11-20T05:32:50.559872+0000 mon.smithi058 (mon.0) 381 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2021-11-20T05:32:51.886 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:51 smithi187 conmon[36358]: audit 2021-11-20T05:32:50.560257+0000 mon.smithi058 (mon.0) 382 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T05:32:51.886 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:51 smithi187 conmon[36358]: audit 2021-11-20T05:32:50.565182+0000 mon.smithi058 (mon.0) 383 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T05:32:51.886 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:51 smithi187 conmon[36358]: audit 2021-11-20T05:32:51.073174+0000 mgr.smithi058.zagdlk (mgr.14164) 137 : audit [DBG] from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:32:51.887 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:51 smithi187 conmon[36358]: audit 2021-11-20T05:32:51.074110+0000 mon.smithi058 (mon.0) 384 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T05:32:51.887 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:51 smithi187 conmon[36358]: cephadm 2021-11-20T05:32:51.075671+0000 mgr.smithi058.zagdlk (mgr.14164) 138 : cephadm [INF] Applying service osd.None on host smithi187... 2021-11-20T05:32:51.887 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:51 smithi187 conmon[36358]: audit 2021-11-20T05:32:51.076014+0000 mon.smithi058 (mon.0) 385 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T05:32:51.888 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:51 smithi187 conmon[36358]: audit 2021-11-20T05:32:51.076882+0000 mon.smithi058 (mon.0) 386 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:32:51.969 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: audit 2021-11-20T05:32:50.559727+0000 mon.smithi058 (mon.0) 380 2021-11-20T05:32:51.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2021-11-20T05:32:51.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: cluster 2021-11-20T05:32:50.559872+0000 mon.smithi058 (mon.0) 381 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2021-11-20T05:32:51.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: audit 2021-11-20T05:32:50.560257+0000 2021-11-20T05:32:51.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: mon.smithi058 (mon.0) 382 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T05:32:51.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: audit 2021-11-20T05:32:50 2021-11-20T05:32:51.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: .565182+0000 mon.smithi058 (mon.0) 383 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T05:32:51.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:32:51.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: 11-20T05:32:51.073174+0000 2021-11-20T05:32:51.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 137 : audit [DBG] 2021-11-20T05:32:51.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:32:51.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: audit 2021-11-20T05:32:51.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: 2021-11-20T05:32:51.074110+0000 mon.smithi058 2021-11-20T05:32:51.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: (mon.0) 384 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T05:32:51.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: 2021-11-20T05:32:51.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: cephadm 2021-11-20T05:32:51.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: 2021-11-20T05:32:51.075671+0000 mgr.smithi058.zagdlk 2021-11-20T05:32:51.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: (mgr.14164) 138 : cephadm [INF] Applying service osd.None on host smithi187... 2021-11-20T05:32:51.975 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: 2021-11-20T05:32:51.975 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: audit 2021-11-20T05:32:51.975 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: 2021-11-20T05:32:51.076014+0000 2021-11-20T05:32:51.975 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: mon.smithi058 (mon.0) 385 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T05:32:51.976 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: 2021-11-20T05:32:51.976 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: audit 2021-11-20T05:32:51.976 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: 2021-11-20T05:32:51.076882+0000 2021-11-20T05:32:51.976 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: mon.smithi058 (mon.0) 386 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:32:51.977 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:51 smithi058 conmon[33537]: 2021-11-20T05:32:52.892 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:52 smithi187 conmon[36358]: cluster 2021-11-20T05:32:50.070198+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T05:32:52.893 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:52 smithi187 conmon[36358]: cluster 2021-11-20T05:32:50.070296+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T05:32:52.893 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:52 smithi187 conmon[36358]: cluster 2021-11-20T05:32:51.565493+0000 mon.smithi058 (mon.0) 387 : cluster [INF] osd.4 [v2:172.21.15.187:6808/3483552299,v1:172.21.15.187:6809/3483552299] boot 2021-11-20T05:32:52.893 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:52 smithi187 conmon[36358]: cluster 2021-11-20T05:32:51.565548+0000 mon.smithi058 (mon.0) 388 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2021-11-20T05:32:52.894 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:52 smithi187 conmon[36358]: audit 2021-11-20T05:32:51.565950+0000 mon.smithi058 (mon.0) 389 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T05:32:52.894 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:52 smithi187 conmon[36358]: cluster 2021-11-20T05:32:52.195643+0000 mgr.smithi058.zagdlk (mgr.14164) 139 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:52.894 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:52 smithi187 conmon[36358]: cephadm 2021-11-20T05:32:52.288287+0000 mgr.smithi058.zagdlk (mgr.14164) 140 : cephadm [INF] Detected new or changed devices on smithi187 2021-11-20T05:32:52.894 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:52 smithi187 conmon[36358]: audit 2021-11-20T05:32:52.293604+0000 mon.smithi058 (mon.0) 390 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:52.894 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:52 smithi187 conmon[36358]: audit 2021-11-20T05:32:52.297523+0000 mon.smithi058 (mon.0) 391 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:52.895 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:52 smithi187 conmon[36358]: audit 2021-11-20T05:32:52.301266+0000 mon.smithi058 (mon.0) 392 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:32:52.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: cluster 2021-11-20T05:32:50.070198 2021-11-20T05:32:52.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: +0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T05:32:52.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: cluster 2021-11-20T05:32:50.070296 2021-11-20T05:32:52.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: +0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T05:32:52.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: cluster 2021-11-20T05:32:52.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: 2021-11-20T05:32:51. 2021-11-20T05:32:52.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: 565493+0000 mon.smithi058 (mon.0) 387 : cluster 2021-11-20T05:32:52.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: [INF] osd.4 [v2:172.21.15.187:6808/3483552299,v1:172.21.15.187:6809/3483552299] boot 2021-11-20T05:32:52.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: cluster 2021-11 2021-11-20T05:32:52.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: -20T05:32:51.565548+0000 2021-11-20T05:32:52.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: mon.smithi058 (mon.0) 388 : cluster 2021-11-20T05:32:52.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: [DBG] osdmap e31: 5 total, 5 up, 5 in 2021-11-20T05:32:52.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: audit 2021 2021-11-20T05:32:52.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: -11-20T05:32:51.565950 2021-11-20T05:32:52.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 389 : audit 2021-11-20T05:32:52.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T05:32:52.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: cluster 2021-11 2021-11-20T05:32:52.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: -20T05:32:52.195643+0000 2021-11-20T05:32:52.975 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 139 : cluster [DBG] 2021-11-20T05:32:52.975 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: pgmap v106: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T05:32:52.975 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: cephadm 2021-11-20T05:32:52 2021-11-20T05:32:52.975 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: .288287+0000 mgr.smithi058.zagdlk (mgr.14164) 140 : cephadm [INF] 2021-11-20T05:32:52.975 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: Detected new or changed devices on smithi187 2021-11-20T05:32:52.976 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: audit 2021-11-20T05:32: 2021-11-20T05:32:52.976 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: 52.293604+0000 mon.smithi058 (mon.0) 390 : audit [INF] 2021-11-20T05:32:52.976 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:52.976 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: audit 2021-11-20T05:32:52.977 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: 2021-11-20T05:32:52.297523+0000 mon.smithi058 (mon. 2021-11-20T05:32:52.977 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: 0) 391 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:52.977 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: audit 2021-11-20 2021-11-20T05:32:52.977 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: T05:32:52.301266+0000 mon.smithi058 (mon 2021-11-20T05:32:52.978 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:52 smithi058 conmon[33537]: .0) 392 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:32:53.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:53 smithi187 conmon[36358]: cluster 2021-11-20T05:32:52.573165+0000 mon.smithi058 (mon.0) 2021-11-20T05:32:53.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:53 smithi187 conmon[36358]: 393 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2021-11-20T05:32:53.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:53 smithi058 conmon[33537]: cluster 2021-11-20T05:32:52.573165+0000 2021-11-20T05:32:53.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:53 smithi058 conmon[33537]: mon.smithi058 (mon.0) 393 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2021-11-20T05:32:54.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:54 smithi187 conmon[36358]: audit 2021-11-20T05:32:53.576335+0000 mon.smithi187 (mon.1) 10 : audit [INF] from='client.? 172.21.15.187:0/1847304897' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8190ded3-7b78-4d71-9c23-e5bc4112fecd"}]: dispatch 2021-11-20T05:32:54.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:54 smithi187 conmon[36358]: audit 2021-11-20T05:32:53.576785+0000 mon.smithi058 (mon.0) 394 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8190ded3-7b78-4d71-9c23-e5bc4112fecd"}]: dispatch 2021-11-20T05:32:54.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:54 smithi187 conmon[36358]: audit 2021-11-20T05:32:53.579940+0000 mon.smithi058 (mon.0) 395 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8190ded3-7b78-4d71-9c23-e5bc4112fecd"}]': finished 2021-11-20T05:32:54.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:54 smithi187 conmon[36358]: cluster 2021-11-20T05:32:53.580000+0000 mon.smithi058 (mon.0) 396 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2021-11-20T05:32:54.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:54 smithi187 conmon[36358]: audit 2021-11-20T05:32:53.580109+0000 mon.smithi058 (mon.0) 397 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T05:32:54.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:54 smithi187 conmon[36358]: 2021-11-20T05:32:54.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:54 smithi187 conmon[36358]: audit 2021-11-20T05:32:54.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:54 smithi187 conmon[36358]: 2021-11-20T05:32:53.998099+0000 mon.smithi187 (mon. 2021-11-20T05:32:54.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:54 smithi187 conmon[36358]: 1) 11 : audit [DBG] from='client.? 172.21.15.187:0/2311367774' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T05:32:54.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:54 smithi187 conmon[36358]: cluster 2021-11 2021-11-20T05:32:54.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:54 smithi187 conmon[36358]: -20T05:32:54.195976+0000 mgr.smithi058.zagdlk 2021-11-20T05:32:54.950 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:54 smithi187 conmon[36358]: (mgr.14164) 141 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:32:54.969 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: audit 2021-11-20T05:32:53.576335+0000 mon.smithi187 (mon.1) 10 : audit [INF] from='client.? 172.21.15.187:0/1847304897' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8190ded3-7b78-4d71-9c23-e5bc4112fecd"}]: dispatch 2021-11-20T05:32:54.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: audit 2021-11-20T05:32:53.576785+0000 mon.smithi058 (mon.0) 394 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8190ded3-7b78-4d71-9c23-e5bc4112fecd"}]: dispatch 2021-11-20T05:32:54.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: audit 2021-11-20T05:32: 2021-11-20T05:32:54.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: 53.579940+0000 mon.smithi058 (mon.0) 395 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8190ded3-7b78-4d71-9c23-e5bc4112fecd"}]': finished 2021-11-20T05:32:54.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: cluster 2021-11 2021-11-20T05:32:54.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: -20T05:32:53.580000+0000 2021-11-20T05:32:54.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: mon.smithi058 (mon.0) 396 : cluster [DBG] 2021-11-20T05:32:54.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: osdmap e33: 6 total, 5 up, 6 in 2021-11-20T05:32:54.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: audit 2021-11-20T05:32:54.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: 2021-11-20T05:32:53.580109+0000 2021-11-20T05:32:54.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: mon.smithi058 (mon.0) 397 : audit [DBG] 2021-11-20T05:32:54.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T05:32:54.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: audit 2021-11-20T05:32: 2021-11-20T05:32:54.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: 53.998099+0000 mon.smithi187 (mon.1) 2021-11-20T05:32:54.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: 11 : audit [DBG] from='client.? 172.21.15.187:0/2311367774' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T05:32:54.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: cluster 2021-11 2021-11-20T05:32:54.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: -20T05:32:54.195976 2021-11-20T05:32:54.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: +0000 mgr.smithi058.zagdlk (mgr.14164) 141 : cluster 2021-11-20T05:32:54.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:54 smithi058 conmon[33537]: [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:32:57.623 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:57 smithi187 conmon[36358]: cluster 2021-11-20T05:32:56.196279+0000 mgr.smithi058.zagdlk (mgr.14164) 142 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:32:57.624 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:57 smithi187 conmon[36358]: audit 2021-11-20T05:32:57.200616+0000 mon.smithi058 (mon.0) 398 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:57.624 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:57 smithi187 conmon[36358]: audit 2021-11-20T05:32:57.203571+0000 mon.smithi058 (mon.0) 399 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:57.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:57 smithi058 conmon[33537]: cluster 2021-11-20T05:32:56.196279+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:32:57.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:57 smithi058 conmon[33537]: 142 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:32:57.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:57 smithi058 conmon[33537]: audit 2021-11-20T05:32:57.200616+0000 mon.smithi058 (mon.0) 398 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:57.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:57 smithi058 conmon[33537]: audit 2021-11-20T05:32 2021-11-20T05:32:57.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:57 smithi058 conmon[33537]: :57.203571+0000 mon.smithi058 (mon.0) 399 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:32:59.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:32:59 smithi187 conmon[36358]: cluster 2021-11-20T05:32:58.196700+0000 mgr.smithi058.zagdlk (mgr.14164) 143 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:32:59.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:59 smithi058 conmon[33537]: cluster 2021-11-20T05:32:58.196700+0000 mgr.smithi058.zagdlk (mgr.14164) 143 : cluster 2021-11-20T05:32:59.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:32:59 smithi058 conmon[33537]: [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:33:00.555 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:00 smithi187 conmon[36358]: audit 2021-11-20T05:33:00.000945+0000 mon.smithi058 (mon.0) 400 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-11-20T05:33:00.555 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:00 smithi187 conmon[36358]: audit 2021 2021-11-20T05:33:00.556 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:00 smithi187 conmon[36358]: -11-20T05:33:00.001896+0000 mon.smithi058 (mon.0) 401 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:33:00.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:00 smithi058 conmon[33537]: audit 2021-11-20T05:33:00.000945+0000 mon.smithi058 (mon.0) 400 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-11-20T05:33:00.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:00 smithi058 conmon[33537]: audit 2021-11-20T05:33:00.001896+0000 mon.smithi058 (mon 2021-11-20T05:33:00.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:00 smithi058 conmon[33537]: .0) 401 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:33:01.522 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:01 smithi187 conmon[36358]: cephadm 2021-11-20T05:33:00.002616+0000 mgr.smithi058.zagdlk (mgr.14164) 144 : cephadm [INF] Deploying daemon osd.5 on smithi187 2021-11-20T05:33:01.523 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:01 smithi187 conmon[36358]: cluster 2021-11-20T 2021-11-20T05:33:01.523 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:01 smithi187 conmon[36358]: 05:33:00.197074+0000 mgr.smithi058.zagdlk (mgr.14164) 145 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:33:01.523 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:01 smithi187 conmon[36358]: audit 2021-11-20T05:33:00.372569+0000 mon.smithi058 (mon.0) 402 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:33:01.523 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:01 smithi187 conmon[36358]: audit 2021-11-20T05:33:00.372795+0000 mon.smithi058 (mon.0) 403 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:33:01.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:01 smithi058 conmon[33537]: cephadm 2021-11-20T05:33:00.002616+0000 mgr.smithi058.zagdlk (mgr.14164) 144 : cephadm [INF] Deploying daemon osd.5 on smithi187 2021-11-20T05:33:01.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:01 smithi058 conmon[33537]: cluster 2021-11-20T05:33:00.197074+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:33:01.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:01 smithi058 conmon[33537]: mgr.14164) 145 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:33:01.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:01 smithi058 conmon[33537]: audit 2021-11-20T05:33:00.372569+0000 mon.smithi058 (mon.0) 402 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:33:01.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:01 smithi058 conmon[33537]: audit 2021-11-20T05:33:00.372795+0000 2021-11-20T05:33:01.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:01 smithi058 conmon[33537]: mon.smithi058 (mon.0) 403 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:33:03.303 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 5 on host 'smithi187' 2021-11-20T05:33:03.304 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:03.301+0000 7f31a7fff700 1 -- 172.21.15.187:0/1863108242 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f31c00fef80 con 0x7f31a805c300 2021-11-20T05:33:03.304 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:03.303+0000 7f31c5a7f700 1 -- 172.21.15.187:0/1863108242 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f31a805c300 msgr2=0x7f31a805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:03.304 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:03.303+0000 7f31c5a7f700 1 --2- 172.21.15.187:0/1863108242 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f31a805c300 0x7f31a805e7c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f31c00f77e0 tx=0x7f31b4009c00).stop 2021-11-20T05:33:03.304 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:03.304+0000 7f31c5a7f700 1 -- 172.21.15.187:0/1863108242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31c00f7980 msgr2=0x7f31c0107490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:03.305 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:03.304+0000 7f31c5a7f700 1 --2- 172.21.15.187:0/1863108242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31c00f7980 0x7f31c0107490 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f31b0009f50 tx=0x7f31b0009b30).stop 2021-11-20T05:33:03.305 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:03.304+0000 7f31c5a7f700 1 -- 172.21.15.187:0/1863108242 shutdown_connections 2021-11-20T05:33:03.305 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:03.304+0000 7f31c5a7f700 1 --2- 172.21.15.187:0/1863108242 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f31a805c300 0x7f31a805e7c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:03.305 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:03.304+0000 7f31c5a7f700 1 --2- 172.21.15.187:0/1863108242 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f31c00f66e0 0x7f31c0106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:03.306 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:03.304+0000 7f31c5a7f700 1 --2- 172.21.15.187:0/1863108242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31c00f7980 0x7f31c0107490 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:03.306 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:03.304+0000 7f31c5a7f700 1 -- 172.21.15.187:0/1863108242 >> 172.21.15.187:0/1863108242 conn(0x7f31c00f1e40 msgr2=0x7f31c00fb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:03.306 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:03.304+0000 7f31c5a7f700 1 -- 172.21.15.187:0/1863108242 shutdown_connections 2021-11-20T05:33:03.306 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:03.304+0000 7f31c5a7f700 1 -- 172.21.15.187:0/1863108242 wait complete. 2021-11-20T05:33:03.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:03 smithi187 conmon[36358]: cluster 2021-11-20T05:33:02.197583+0000 mgr.smithi058.zagdlk (mgr.14164) 146 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:33:03.707 DEBUG:teuthology.orchestra.run.smithi187:osd.5> sudo journalctl -f -n 0 -u ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.5.service 2021-11-20T05:33:03.711 INFO:tasks.cephadm:Setting up client nodes... 2021-11-20T05:33:03.711 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-11-20T05:33:03.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:03 smithi058 conmon[33537]: cluster 2021-11-20T05:33:02.197583+0000 mgr.smithi058.zagdlk (mgr.14164) 146 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:33:03.741 INFO:journalctl@ceph.osd.5.smithi187.stdout:-- Logs begin at Sat 2021-11-20 05:17:59 UTC. -- 2021-11-20T05:33:04.529 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:04 smithi058 conmon[33537]: audit 2021-11-20T05:33:03.291134+0000 mon.smithi058 (mon.0) 404 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:33:04.529 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:04 smithi058 conmon[33537]: audit 2021-11-20T05:33:03.295557+0000 mon.smithi058 (mon.0) 405 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:33:04.530 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:04 smithi058 conmon[33537]: audit 2021-11-20T05:33:03.300805+0000 mon.smithi058 (mon.0) 406 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:33:04.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:04 smithi187 conmon[36358]: audit 2021-11-20T05:33:03.291134+0000 2021-11-20T05:33:04.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:04 smithi187 conmon[36358]: mon.smithi058 (mon.0) 404 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:33:04.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:04 smithi187 conmon[36358]: audit 2021-11-20T05:33:03.295557+0000 mon.smithi058 (mon.0) 405 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:33:04.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:04 smithi187 conmon[36358]: audit 2021-11-20T05:33:03.300805+0000 mon.smithi058 (mon.0) 406 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:33:05.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:05 smithi058 conmon[33537]: cluster 2021-11-20T05:33:04.197957+0000 mgr.smithi058.zagdlk (mgr.14164) 147 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:33:05.593 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.592+0000 7f564738c700 1 -- 172.21.15.58:0/2828997641 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f56400f4780 msgr2=0x7f56400f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:05.594 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.592+0000 7f564738c700 1 --2- 172.21.15.58:0/2828997641 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f56400f4780 0x7f56400f4c00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f563c004660 tx=0x7f563c009b30).stop 2021-11-20T05:33:05.594 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.592+0000 7f564738c700 1 -- 172.21.15.58:0/2828997641 shutdown_connections 2021-11-20T05:33:05.594 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.592+0000 7f564738c700 1 --2- 172.21.15.58:0/2828997641 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f56400f4780 0x7f56400f4c00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:05.594 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.592+0000 7f564738c700 1 --2- 172.21.15.58:0/2828997641 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56400f7d90 0x7f56400f81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:05.594 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.592+0000 7f564738c700 1 -- 172.21.15.58:0/2828997641 >> 172.21.15.58:0/2828997641 conn(0x7f56400f04a0 msgr2=0x7f56400f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:05.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.592+0000 7f564738c700 1 -- 172.21.15.58:0/2828997641 shutdown_connections 2021-11-20T05:33:05.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.592+0000 7f564738c700 1 -- 172.21.15.58:0/2828997641 wait complete. 2021-11-20T05:33:05.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.592+0000 7f564738c700 1 Processor -- start 2021-11-20T05:33:05.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.593+0000 7f564738c700 1 -- start start 2021-11-20T05:33:05.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.593+0000 7f564738c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f56400f4780 0x7f56401070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:05.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.593+0000 7f564738c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56400f7d90 0x7f56401075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:05.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.593+0000 7f564738c700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5640107b60 con 0x7f56400f4780 2021-11-20T05:33:05.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.593+0000 7f564738c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5640107ca0 con 0x7f56400f7d90 2021-11-20T05:33:05.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.593+0000 7f5645128700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f56400f4780 0x7f56401070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:05.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.593+0000 7f5645128700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f56400f4780 0x7f56401070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40192/0 (socket says 172.21.15.58:40192) 2021-11-20T05:33:05.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.593+0000 7f5645128700 1 -- 172.21.15.58:0/239966291 learned_addr learned my addr 172.21.15.58:0/239966291 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:05.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.593+0000 7f5644927700 1 --2- 172.21.15.58:0/239966291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56400f7d90 0x7f56401075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:05.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.593+0000 7f5645128700 1 -- 172.21.15.58:0/239966291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56400f7d90 msgr2=0x7f56401075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:05.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.593+0000 7f5645128700 1 --2- 172.21.15.58:0/239966291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56400f7d90 0x7f56401075b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:05.598 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.593+0000 7f5645128700 1 -- 172.21.15.58:0/239966291 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c005040 con 0x7f56400f4780 2021-11-20T05:33:05.598 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.593+0000 7f5645128700 1 --2- 172.21.15.58:0/239966291 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f56400f4780 0x7f56401070c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f563800d7f0 tx=0x7f5638004e80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:05.598 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.593+0000 7f5635ffb700 1 -- 172.21.15.58:0/239966291 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f563800f490 con 0x7f56400f4780 2021-11-20T05:33:05.598 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.593+0000 7f5635ffb700 1 -- 172.21.15.58:0/239966291 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f563800f040 con 0x7f56400f4780 2021-11-20T05:33:05.599 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.593+0000 7f564738c700 1 -- 172.21.15.58:0/239966291 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5640108640 con 0x7f56400f4780 2021-11-20T05:33:05.599 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.593+0000 7f5635ffb700 1 -- 172.21.15.58:0/239966291 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56380139b0 con 0x7f56400f4780 2021-11-20T05:33:05.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.594+0000 7f564738c700 1 -- 172.21.15.58:0/239966291 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5640108c00 con 0x7f56400f4780 2021-11-20T05:33:05.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.595+0000 7f564738c700 1 -- 172.21.15.58:0/239966291 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5640045ad0 con 0x7f56400f4780 2021-11-20T05:33:05.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.595+0000 7f5635ffb700 1 -- 172.21.15.58:0/239966291 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f563801a030 con 0x7f56400f4780 2021-11-20T05:33:05.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.595+0000 7f5635ffb700 1 --2- 172.21.15.58:0/239966291 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f562c05c2b0 0x7f562c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:05.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.595+0000 7f5635ffb700 1 -- 172.21.15.58:0/239966291 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4235+0+0 (secure 0 0 0) 0x7f5638075840 con 0x7f56400f4780 2021-11-20T05:33:05.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.595+0000 7f5644927700 1 --2- 172.21.15.58:0/239966291 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f562c05c2b0 0x7f562c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:05.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.596+0000 7f5644927700 1 --2- 172.21.15.58:0/239966291 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f562c05c2b0 0x7f562c05e770 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f563c009f50 tx=0x7f563c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:05.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.600+0000 7f5635ffb700 1 -- 172.21.15.58:0/239966291 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5638048980 con 0x7f56400f4780 2021-11-20T05:33:05.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:05 smithi187 conmon[36358]: cluster 2021-11-20T05:33:04.197957+0000 mgr.smithi058.zagdlk (mgr.14164) 147 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:33:05.826 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.826+0000 7f564738c700 1 -- 172.21.15.58:0/239966291 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f5640027a00 con 0x7f56400f4780 2021-11-20T05:33:05.829 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.830+0000 7f5635ffb700 1 -- 172.21.15.58:0/239966291 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f563804bfa0 con 0x7f56400f4780 2021-11-20T05:33:05.830 INFO:teuthology.orchestra.run.smithi058.stdout:[client.0] 2021-11-20T05:33:05.830 INFO:teuthology.orchestra.run.smithi058.stdout: key = AQCRiJhhd2dbMRAAth/7GwAQm2zOHnT+SNmA4g== 2021-11-20T05:33:05.831 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.832+0000 7f564738c700 1 -- 172.21.15.58:0/239966291 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f562c05c2b0 msgr2=0x7f562c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:05.832 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.832+0000 7f564738c700 1 --2- 172.21.15.58:0/239966291 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f562c05c2b0 0x7f562c05e770 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f563c009f50 tx=0x7f563c009b30).stop 2021-11-20T05:33:05.832 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.832+0000 7f564738c700 1 -- 172.21.15.58:0/239966291 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f56400f4780 msgr2=0x7f56401070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:05.832 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.832+0000 7f564738c700 1 --2- 172.21.15.58:0/239966291 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f56400f4780 0x7f56401070c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f563800d7f0 tx=0x7f5638004e80).stop 2021-11-20T05:33:05.832 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.832+0000 7f564738c700 1 -- 172.21.15.58:0/239966291 shutdown_connections 2021-11-20T05:33:05.833 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.832+0000 7f564738c700 1 --2- 172.21.15.58:0/239966291 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f562c05c2b0 0x7f562c05e770 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:05.833 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.832+0000 7f564738c700 1 --2- 172.21.15.58:0/239966291 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f56400f4780 0x7f56401070c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:05.833 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.832+0000 7f564738c700 1 --2- 172.21.15.58:0/239966291 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56400f7d90 0x7f56401075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:05.833 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.832+0000 7f564738c700 1 -- 172.21.15.58:0/239966291 >> 172.21.15.58:0/239966291 conn(0x7f56400f04a0 msgr2=0x7f56400fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:05.834 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.832+0000 7f564738c700 1 -- 172.21.15.58:0/239966291 shutdown_connections 2021-11-20T05:33:05.834 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:05.832+0000 7f564738c700 1 -- 172.21.15.58:0/239966291 wait complete. 2021-11-20T05:33:06.219 DEBUG:teuthology.orchestra.run.smithi058:> set -ex 2021-11-20T05:33:06.220 DEBUG:teuthology.orchestra.run.smithi058:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2021-11-20T05:33:06.220 DEBUG:teuthology.orchestra.run.smithi058:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-11-20T05:33:06.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-11-20T05:33:06.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:06 smithi058 conmon[33537]: audit 2021-11-20T05:33:05.827911+0000 mon.smithi058 2021-11-20T05:33:06.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:06 smithi058 conmon[33537]: (mon.0) 407 : audit [INF] from='client.? 172.21.15.58:0/239966291' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T05:33:06.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:06 smithi058 conmon[33537]: audit 2021-11-20T05:33:05.831017+0000 mon.smithi058 (mon.0) 408 : audit [INF] from='client.? 172.21.15.58:0/239966291' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-20T05:33:06.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:06 smithi058 conmon[33537]: audit 2021-11-20T05:33:06.117118+0000 mon.smithi058 (mon.0) 409 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:33:06.614 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:06 smithi187 conmon[36358]: audit 2021-11-20T05:33:05.827911+0000 mon.smithi058 (mon.0 2021-11-20T05:33:06.615 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:06 smithi187 conmon[36358]: ) 407 : audit [INF] from='client.? 172.21.15.58:0/239966291' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T05:33:06.615 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:06 smithi187 conmon[36358]: audit 2021-11-20T05:33:05.831017+0000 mon.smithi058 (mon.0) 408 : audit [INF] from='client.? 172.21.15.58:0/239966291' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-20T05:33:06.616 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:06 smithi187 conmon[36358]: audit 2021-11-20T05:33:06.117118+0000 mon.smithi058 (mon.0) 409 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:33:07.128 INFO:journalctl@ceph.osd.5.smithi187.stdout:Nov 20 05:33:07 smithi187 conmon[52734]: debug 2021-11-20T05:33:07.118+0000 7f11b2496080 -1 osd.5 0 log_to_monitors {default=true} 2021-11-20T05:33:07.585 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:07 smithi187 conmon[36358]: cluster 2021-11-20T05:33:06.198267+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:33:07.585 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:07 smithi187 conmon[36358]: mgr.14164) 148 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:33:07.585 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:07 smithi187 conmon[36358]: audit 2021-11-20T05:33:07.121496+0000 mon.smithi187 (mon.1) 12 : audit 2021-11-20T05:33:07.586 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:07 smithi187 conmon[36358]: [INF] from='osd.5 [v2:172.21.15.187:6816/566077514,v1:172.21.15.187:6817/566077514]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-20T05:33:07.586 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:07 smithi187 conmon[36358]: audit 2021-11-20T 2021-11-20T05:33:07.586 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:07 smithi187 conmon[36358]: 05:33:07.121929+0000 mon.smithi058 (mon.0) 410 : audit [INF] 2021-11-20T05:33:07.586 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:07 smithi187 conmon[36358]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-20T05:33:07.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:07 smithi058 conmon[33537]: cluster 2021-11-20T05:33:06 2021-11-20T05:33:07.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:07 smithi058 conmon[33537]: .198267+0000 mgr.smithi058.zagdlk (mgr.14164) 148 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:33:07.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:07 smithi058 conmon[33537]: audit 2021-11-20T 2021-11-20T05:33:07.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:07 smithi058 conmon[33537]: 05:33:07.121496+0000 mon.smithi187 (mon.1) 12 : audit 2021-11-20T05:33:07.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:07 smithi058 conmon[33537]: [INF] from='osd.5 [v2:172.21.15.187:6816/566077514,v1:172.21.15.187:6817/566077514]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-20T05:33:07.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:07 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:33:07.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:07 smithi058 conmon[33537]: -20T05:33:07. 2021-11-20T05:33:07.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:07 smithi058 conmon[33537]: 121929+0000 mon.smithi058 (mon.0) 2021-11-20T05:33:07.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:07 smithi058 conmon[33537]: 410 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-20T05:33:08.525 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.523+0000 7f4d88d82700 1 -- 172.21.15.187:0/2649474424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d840f6550 msgr2=0x7f4d840f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:08.525 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.523+0000 7f4d88d82700 1 --2- 172.21.15.187:0/2649474424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d840f6550 0x7f4d840f69b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f4d6c004660 tx=0x7f4d6c009b30).stop 2021-11-20T05:33:08.526 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.523+0000 7f4d88d82700 1 -- 172.21.15.187:0/2649474424 shutdown_connections 2021-11-20T05:33:08.526 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.523+0000 7f4d88d82700 1 --2- 172.21.15.187:0/2649474424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d840f6550 0x7f4d840f69b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:08.526 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.523+0000 7f4d88d82700 1 --2- 172.21.15.187:0/2649474424 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4d840f5bb0 0x7f4d840f5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:08.526 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.523+0000 7f4d88d82700 1 -- 172.21.15.187:0/2649474424 >> 172.21.15.187:0/2649474424 conn(0x7f4d840f04c0 msgr2=0x7f4d840f28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:08.527 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.523+0000 7f4d88d82700 1 -- 172.21.15.187:0/2649474424 shutdown_connections 2021-11-20T05:33:08.527 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.523+0000 7f4d88d82700 1 -- 172.21.15.187:0/2649474424 wait complete. 2021-11-20T05:33:08.527 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.524+0000 7f4d88d82700 1 Processor -- start 2021-11-20T05:33:08.527 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.524+0000 7f4d88d82700 1 -- start start 2021-11-20T05:33:08.527 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.524+0000 7f4d88d82700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d840f5bb0 0x7f4d84109c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:08.528 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.524+0000 7f4d88d82700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4d840f6550 0x7f4d8410c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:08.528 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.524+0000 7f4d88d82700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d840f5680 con 0x7f4d840f6550 2021-11-20T05:33:08.528 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.524+0000 7f4d88d82700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d840f57c0 con 0x7f4d840f5bb0 2021-11-20T05:33:08.528 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.524+0000 7f4d8259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d840f5bb0 0x7f4d84109c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:08.529 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.524+0000 7f4d8259c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d840f5bb0 0x7f4d84109c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:44518/0 (socket says 172.21.15.187:44518) 2021-11-20T05:33:08.529 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.524+0000 7f4d8259c700 1 -- 172.21.15.187:0/3231496501 learned_addr learned my addr 172.21.15.187:0/3231496501 (peer_addr_for_me v2:172.21.15.187:0/0) 2021-11-20T05:33:08.529 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.524+0000 7f4d79d9b700 1 --2- 172.21.15.187:0/3231496501 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4d840f6550 0x7f4d8410c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:08.529 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.525+0000 7f4d8259c700 1 -- 172.21.15.187:0/3231496501 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4d840f6550 msgr2=0x7f4d8410c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:08.529 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.525+0000 7f4d8259c700 1 --2- 172.21.15.187:0/3231496501 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4d840f6550 0x7f4d8410c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:08.530 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.525+0000 7f4d8259c700 1 -- 172.21.15.187:0/3231496501 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6c005040 con 0x7f4d840f5bb0 2021-11-20T05:33:08.530 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.525+0000 7f4d8259c700 1 --2- 172.21.15.187:0/3231496501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d840f5bb0 0x7f4d84109c70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f4d74002700 tx=0x7f4d74009e00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:08.530 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.525+0000 7f4d7b7fe700 1 -- 172.21.15.187:0/3231496501 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d740079c0 con 0x7f4d840f5bb0 2021-11-20T05:33:08.530 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.525+0000 7f4d88d82700 1 -- 172.21.15.187:0/3231496501 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8410c720 con 0x7f4d840f5bb0 2021-11-20T05:33:08.531 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.525+0000 7f4d7b7fe700 1 -- 172.21.15.187:0/3231496501 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d7400f040 con 0x7f4d840f5bb0 2021-11-20T05:33:08.531 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.525+0000 7f4d7b7fe700 1 -- 172.21.15.187:0/3231496501 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d74014850 con 0x7f4d840f5bb0 2021-11-20T05:33:08.531 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.525+0000 7f4d88d82700 1 -- 172.21.15.187:0/3231496501 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8410ccb0 con 0x7f4d840f5bb0 2021-11-20T05:33:08.531 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.526+0000 7f4d88d82700 1 -- 172.21.15.187:0/3231496501 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d840f4150 con 0x7f4d840f5bb0 2021-11-20T05:33:08.532 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.526+0000 7f4d7b7fe700 1 -- 172.21.15.187:0/3231496501 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4d7401b060 con 0x7f4d840f5bb0 2021-11-20T05:33:08.533 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.526+0000 7f4d7b7fe700 1 --2- 172.21.15.187:0/3231496501 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d7005c2b0 0x7f4d7005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:08.533 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.527+0000 7f4d7b7fe700 1 -- 172.21.15.187:0/3231496501 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4256+0+0 (secure 0 0 0) 0x7f4d74075940 con 0x7f4d840f5bb0 2021-11-20T05:33:08.533 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.527+0000 7f4d79d9b700 1 --2- 172.21.15.187:0/3231496501 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d7005c2b0 0x7f4d7005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:08.533 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.527+0000 7f4d79d9b700 1 --2- 172.21.15.187:0/3231496501 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d7005c2b0 0x7f4d7005e770 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f4d6c009f50 tx=0x7f4d6c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:08.534 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.530+0000 7f4d7b7fe700 1 -- 172.21.15.187:0/3231496501 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d74048a70 con 0x7f4d840f5bb0 2021-11-20T05:33:08.744 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.744+0000 7f4d88d82700 1 -- 172.21.15.187:0/3231496501 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f4d84045ad0 con 0x7f4d840f5bb0 2021-11-20T05:33:08.748 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.749+0000 7f4d7b7fe700 1 -- 172.21.15.187:0/3231496501 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f4d7404c090 con 0x7f4d840f5bb0 2021-11-20T05:33:08.748 INFO:teuthology.orchestra.run.smithi187.stdout:[client.1] 2021-11-20T05:33:08.749 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQCUiJhh7fB5LBAAcreyPOYPz38Z/7BA92nQlA== 2021-11-20T05:33:08.751 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.751+0000 7f4d88d82700 1 -- 172.21.15.187:0/3231496501 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d7005c2b0 msgr2=0x7f4d7005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:08.751 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.751+0000 7f4d88d82700 1 --2- 172.21.15.187:0/3231496501 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d7005c2b0 0x7f4d7005e770 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f4d6c009f50 tx=0x7f4d6c009b30).stop 2021-11-20T05:33:08.751 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.751+0000 7f4d88d82700 1 -- 172.21.15.187:0/3231496501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d840f5bb0 msgr2=0x7f4d84109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:08.752 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.751+0000 7f4d88d82700 1 --2- 172.21.15.187:0/3231496501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d840f5bb0 0x7f4d84109c70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f4d74002700 tx=0x7f4d74009e00).stop 2021-11-20T05:33:08.752 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.751+0000 7f4d88d82700 1 -- 172.21.15.187:0/3231496501 shutdown_connections 2021-11-20T05:33:08.753 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.751+0000 7f4d88d82700 1 --2- 172.21.15.187:0/3231496501 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d7005c2b0 0x7f4d7005e770 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:08.753 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.751+0000 7f4d88d82700 1 --2- 172.21.15.187:0/3231496501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d840f5bb0 0x7f4d84109c70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:08.754 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.751+0000 7f4d88d82700 1 --2- 172.21.15.187:0/3231496501 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4d840f6550 0x7f4d8410c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:08.754 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.751+0000 7f4d88d82700 1 -- 172.21.15.187:0/3231496501 >> 172.21.15.187:0/3231496501 conn(0x7f4d840f04c0 msgr2=0x7f4d840fa450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:08.754 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.751+0000 7f4d88d82700 1 -- 172.21.15.187:0/3231496501 shutdown_connections 2021-11-20T05:33:08.755 INFO:teuthology.orchestra.run.smithi187.stderr:2021-11-20T05:33:08.751+0000 7f4d88d82700 1 -- 172.21.15.187:0/3231496501 wait complete. 2021-11-20T05:33:09.143 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2021-11-20T05:33:09.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2021-11-20T05:33:09.144 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2021-11-20T05:33:09.183 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph osd dump --format=json 2021-11-20T05:33:09.283 INFO:journalctl@ceph.osd.5.smithi187.stdout:Nov 20 05:33:09 smithi187 conmon[52734]: debug 2021-11-20T05:33:09.124+0000 7f11a8a07700 -1 osd.5 0 waiting for initial osdmap 2021-11-20T05:33:09.284 INFO:journalctl@ceph.osd.5.smithi187.stdout:Nov 20 05:33:09 smithi187 conmon[52734]: debug 2021-11-20T05:33:09.134+0000 7f11a3980700 -1 osd.5 35 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T05:33:09.284 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:09 smithi187 conmon[36358]: audit 2021-11-20T05:33:08.120772+0000 mon.smithi058 ( 2021-11-20T05:33:09.285 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:09 smithi187 conmon[36358]: mon.0) 411 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-11-20T05:33:09.285 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:09 smithi187 conmon[36358]: cluster 2021-11-20T05:33:08.120830+0000 mon.smithi058 (mon.0) 412 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2021-11-20T05:33:09.285 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:09 smithi187 conmon[36358]: audit 2021-11-20T05:33:08.121171+0000 mon.smithi058 (mon.0) 413 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T05:33:09.285 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:09 smithi187 conmon[36358]: audit 2021-11-20T05:33:08.121382+0000 mon.smithi187 (mon.1) 13 : audit [INF] from='osd.5 [v2:172.21.15.187:6816/566077514,v1:172.21.15.187:6817/566077514]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2021-11-20T05:33:09.285 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:09 smithi187 conmon[36358]: audit 2021-11-20T05:33:08.121647+0000 mon.smithi058 (mon.0) 414 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2021-11-20T05:33:09.286 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:09 smithi187 conmon[36358]: cluster 2021-11-20T05:33:08.198728+0000 mgr.smithi058.zagdlk (mgr.14164) 149 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:33:09.286 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:09 smithi187 conmon[36358]: audit 2021-11-20T05:33:08.745678+0000 mon.smithi187 (mon.1) 14 : audit [INF] from='client.? 172.21.15.187:0/3231496501' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T05:33:09.286 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:09 smithi187 conmon[36358]: audit 2021-11-20T05:33:08.746038+0000 mon.smithi058 (mon.0) 415 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T05:33:09.286 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:09 smithi187 conmon[36358]: audit 2021-11-20T05:33:08.749433+0000 mon.smithi058 (mon.0) 416 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-20T05:33:09.287 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:09 smithi187 conmon[36358]: audit 2021-11-20T05:33:08.801754+0000 mon.smithi058 (mon.0) 417 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:33:09.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:09 smithi058 conmon[33537]: audit 2021-11-20T05:33:08.120772+0000 mon.smithi058 ( 2021-11-20T05:33:09.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:09 smithi058 conmon[33537]: mon.0) 411 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-11-20T05:33:09.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:09 smithi058 conmon[33537]: cluster 2021-11-20T05:33:08.120830+0000 mon.smithi058 (mon.0) 412 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2021-11-20T05:33:09.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:09 smithi058 conmon[33537]: audit 2021-11-20T05:33:08.121171+0000 mon.smithi058 (mon.0) 413 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T05:33:09.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:09 smithi058 conmon[33537]: audit 2021-11-20T05:33:08.121382+0000 mon.smithi187 (mon.1) 13 : audit [INF] from='osd.5 [v2:172.21.15.187:6816/566077514,v1:172.21.15.187:6817/566077514]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2021-11-20T05:33:09.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:09 smithi058 conmon[33537]: audit 2021-11-20T05:33:08.121647+0000 mon.smithi058 (mon.0) 414 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2021-11-20T05:33:09.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:09 smithi058 conmon[33537]: cluster 2021-11-20T05:33:08.198728+0000 mgr.smithi058.zagdlk (mgr.14164) 149 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:33:09.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:09 smithi058 conmon[33537]: audit 2021-11-20T05:33:08.745678+0000 mon.smithi187 (mon.1) 14 : audit [INF] from='client.? 172.21.15.187:0/3231496501' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T05:33:09.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:09 smithi058 conmon[33537]: audit 2021-11-20T05:33:08.746038+0000 mon.smithi058 (mon.0) 415 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T05:33:09.472 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:09 smithi058 conmon[33537]: audit 2021-11-20T05:33:08.749433+0000 mon.smithi058 (mon.0) 416 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-20T05:33:09.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:09 smithi058 conmon[33537]: audit 2021-11-20T05:33:09.473 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:09 smithi058 conmon[33537]: 2021-11-20T05:33:08 2021-11-20T05:33:09.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:09 smithi058 conmon[33537]: .801754+0000 mon.smithi058 (mon.0) 417 : 2021-11-20T05:33:09.474 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:09 smithi058 conmon[33537]: audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:33:09.633 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:10.219 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:10 smithi058 conmon[33537]: cluster 2021-11-20T05:33:08.140345+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T05:33:10.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:10 smithi058 conmon[33537]: cluster 2021-11-20T05:33:08.140439+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T05:33:10.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:10 smithi058 conmon[33537]: audit 2021-11-20T05:33:09.120931+0000 mon.smithi058 (mon.0) 418 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2021-11-20T05:33:10.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:10 smithi058 conmon[33537]: cluster 2021-11-20T05:33:09.120993+0000 mon.smithi058 (mon.0) 419 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2021-11-20T05:33:10.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:10 smithi058 conmon[33537]: audit 2021-11-20T05:33:09.121338+0000 mon.smithi058 (mon.0) 420 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T05:33:10.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:10 smithi058 conmon[33537]: audit 2021-11-20T05:33:09.124843+0000 mon.smithi058 (mon.0) 421 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T05:33:10.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:10 smithi058 conmon[33537]: audit 2021-11-20T05:33:09.859001+0000 mon.smithi058 (mon.0) 422 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:33:10.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:10 smithi187 conmon[36358]: cluster 2021-11-20T05:33:08.140345+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T05:33:10.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:10 smithi187 conmon[36358]: cluster 2021-11-20T05:33:08.140439+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T05:33:10.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:10 smithi187 conmon[36358]: audit 2021-11-20T05:33:09.120931+0000 mon.smithi058 (mon.0) 418 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2021-11-20T05:33:10.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:10 smithi187 conmon[36358]: cluster 2021-11-20T05:33:09.120993+0000 mon.smithi058 (mon.0) 419 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2021-11-20T05:33:10.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:10 smithi187 conmon[36358]: audit 2021-11-20T05:33:09.121338+0000 mon.smithi058 (mon.0) 420 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T05:33:10.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:10 smithi187 conmon[36358]: audit 2021-11-20T05:33:09.124843+0000 mon.smithi058 (mon.0) 421 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T05:33:10.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:10 smithi187 conmon[36358]: audit 2021-11-20T05:33:09.859001+0000 mon.smithi058 (mon.0) 422 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:33:10.972 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.970+0000 7f50e1e76700 1 -- 172.21.15.58:0/1100742317 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f50dc0f9980 msgr2=0x7f50dc0fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:10.972 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.970+0000 7f50e1e76700 1 --2- 172.21.15.58:0/1100742317 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f50dc0f9980 0x7f50dc0fbd70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f50d0004660 tx=0x7f50d0009b30).stop 2021-11-20T05:33:10.972 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.970+0000 7f50e1e76700 1 -- 172.21.15.58:0/1100742317 shutdown_connections 2021-11-20T05:33:10.973 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.970+0000 7f50e1e76700 1 --2- 172.21.15.58:0/1100742317 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f50dc0f9980 0x7f50dc0fbd70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:10.973 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.970+0000 7f50e1e76700 1 --2- 172.21.15.58:0/1100742317 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50dc060810 0x7f50dc0f9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:10.973 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.970+0000 7f50e1e76700 1 -- 172.21.15.58:0/1100742317 >> 172.21.15.58:0/1100742317 conn(0x7f50dc0f0e20 msgr2=0x7f50dc0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:10.973 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.970+0000 7f50e1e76700 1 -- 172.21.15.58:0/1100742317 shutdown_connections 2021-11-20T05:33:10.974 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.970+0000 7f50e1e76700 1 -- 172.21.15.58:0/1100742317 wait complete. 2021-11-20T05:33:10.974 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.971+0000 7f50e1e76700 1 Processor -- start 2021-11-20T05:33:10.974 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.971+0000 7f50e1e76700 1 -- start start 2021-11-20T05:33:10.974 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.971+0000 7f50e1e76700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50dc060810 0x7f50dc0f6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:10.974 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.971+0000 7f50e1e76700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f50dc0f9980 0x7f50dc0f5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:10.975 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.971+0000 7f50e1e76700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc0f5860 con 0x7f50dc0f9980 2021-11-20T05:33:10.975 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.971+0000 7f50e1e76700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc0f59a0 con 0x7f50dc060810 2021-11-20T05:33:10.975 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.971+0000 7f50daffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f50dc0f9980 0x7f50dc0f5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:10.976 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.971+0000 7f50daffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f50dc0f9980 0x7f50dc0f5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40206/0 (socket says 172.21.15.58:40206) 2021-11-20T05:33:10.976 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.971+0000 7f50daffd700 1 -- 172.21.15.58:0/1195848756 learned_addr learned my addr 172.21.15.58:0/1195848756 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:10.976 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.971+0000 7f50db7fe700 1 --2- 172.21.15.58:0/1195848756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50dc060810 0x7f50dc0f6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:10.976 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.971+0000 7f50daffd700 1 -- 172.21.15.58:0/1195848756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50dc060810 msgr2=0x7f50dc0f6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:10.977 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.971+0000 7f50daffd700 1 --2- 172.21.15.58:0/1195848756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50dc060810 0x7f50dc0f6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:10.977 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.971+0000 7f50daffd700 1 -- 172.21.15.58:0/1195848756 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d0005040 con 0x7f50dc0f9980 2021-11-20T05:33:10.977 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.972+0000 7f50daffd700 1 --2- 172.21.15.58:0/1195848756 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f50dc0f9980 0x7f50dc0f5300 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f50d0004e30 tx=0x7f50d0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:10.977 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.972+0000 7f50c3fff700 1 -- 172.21.15.58:0/1195848756 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50d0018070 con 0x7f50dc0f9980 2021-11-20T05:33:10.978 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.972+0000 7f50e1e76700 1 -- 172.21.15.58:0/1195848756 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50dc0f5c20 con 0x7f50dc0f9980 2021-11-20T05:33:10.978 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.972+0000 7f50c3fff700 1 -- 172.21.15.58:0/1195848756 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50d0013410 con 0x7f50dc0f9980 2021-11-20T05:33:10.978 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.972+0000 7f50e1e76700 1 -- 172.21.15.58:0/1195848756 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50dc0f6180 con 0x7f50dc0f9980 2021-11-20T05:33:10.979 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.972+0000 7f50c3fff700 1 -- 172.21.15.58:0/1195848756 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50d001de20 con 0x7f50dc0f9980 2021-11-20T05:33:10.981 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.973+0000 7f50e1e76700 1 -- 172.21.15.58:0/1195848756 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50dc045ad0 con 0x7f50dc0f9980 2021-11-20T05:33:10.981 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.973+0000 7f50c3fff700 1 -- 172.21.15.58:0/1195848756 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f50d001d420 con 0x7f50dc0f9980 2021-11-20T05:33:10.981 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.974+0000 7f50c3fff700 1 --2- 172.21.15.58:0/1195848756 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f50c405c2a0 0x7f50c405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:10.982 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.974+0000 7f50c3fff700 1 -- 172.21.15.58:0/1195848756 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f50d0077e90 con 0x7f50dc0f9980 2021-11-20T05:33:10.982 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.974+0000 7f50db7fe700 1 --2- 172.21.15.58:0/1195848756 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f50c405c2a0 0x7f50c405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:10.982 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.974+0000 7f50db7fe700 1 --2- 172.21.15.58:0/1195848756 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f50c405c2a0 0x7f50c405e760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f50dc0f6c10 tx=0x7f50cc00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:10.982 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:10.977+0000 7f50c3fff700 1 -- 172.21.15.58:0/1195848756 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50d004aec0 con 0x7f50dc0f9980 2021-11-20T05:33:11.144 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:11.143+0000 7f50e1e76700 1 -- 172.21.15.58:0/1195848756 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50dc027a00 con 0x7f50dc0f9980 2021-11-20T05:33:11.144 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:11.144+0000 7f50c3fff700 1 -- 172.21.15.58:0/1195848756 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+10886 (secure 0 0 0) 0x7f50d004e4e0 con 0x7f50dc0f9980 2021-11-20T05:33:11.146 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:33:11.146 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":36,"fsid":"b33f8c4a-49c2-11ec-8c2c-001a4aab830c","created":"2021-11-20T05:28:54.787342+0000","modified":"2021-11-20T05:33:10.122001+0000","last_up_change":"2021-11-20T05:33:10.122001+0000","last_in_change":"2021-11-20T05:32:53.577246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T05:32:27.068478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"e2f5103a-e0d2-4bc3-88e1-f28420c98071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6802","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6803","nonce":2375873844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6804","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6805","nonce":2375873844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6808","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6809","nonce":2375873844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6806","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6807","nonce":2375873844}]},"public_addr":"172.21.15.58:6803/2375873844","cluster_addr":"172.21.15.58:6805/2375873844","heartbeat_back_addr":"172.21.15.58:6809/2375873844","heartbeat_front_addr":"172.21.15.58:6807/2375873844","state":["exists","up"]},{"osd":1,"uuid":"e0243e40-c76e-4c68-8ac0-8f555035e948","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6810","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6811","nonce":3875723869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6812","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6813","nonce":3875723869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6816","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6817","nonce":3875723869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6814","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6815","nonce":3875723869}]},"public_addr":"172.21.15.58:6811/3875723869","cluster_addr":"172.21.15.58:6813/3875723869","heartbeat_back_addr":"172.21.15.58:6817/3875723869","heartbeat_front_addr":"172.21.15.58:6815/3875723869","state":["exists","up"]},{"osd":2,"uuid":"9b35ec54-f60f-49c5-ab1b-d2c5b5eb72e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6818","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6819","nonce":745015076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6820","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6821","nonce":745015076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6824","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6825","nonce":745015076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6822","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6823","nonce":745015076}]},"public_addr":"172.21.15.58:6819/745015076","cluster_addr":"172.21.15.58:6821/745015076","heartbeat_back_addr":"172.21.15.58:6825/745015076","heartbeat_front_addr":"172.21.15.58:6823/745015076","state":["exists","up"]},{"osd":3,"uuid":"8390f146-b7e1-4039-83d3-7ed1c832dc8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6801","nonce":1941886381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6803","nonce":1941886381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6807","nonce":1941886381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6805","nonce":1941886381}]},"public_addr":"172.21.15.187:6801/1941886381","cluster_addr":"172.21.15.187:6803/1941886381","heartbeat_back_addr":"172.21.15.187:6807/1941886381","heartbeat_front_addr":"172.21.15.187:6805/1941886381","state":["exists","up"]},{"osd":4,"uuid":"89dc0c2f-c52c-48ca-b3ab-792e299d494d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6809","nonce":3483552299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6811","nonce":3483552299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6815","nonce":3483552299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6813","nonce":3483552299}]},"public_addr":"172.21.15.187:6809/3483552299","cluster_addr":"172.21.15.187:6811/3483552299","heartbeat_back_addr":"172.21.15.187:6815/3483552299","heartbeat_front_addr":"172.21.15.187:6813/3483552299","state":["exists","up"]},{"osd":5,"uuid":"8190ded3-7b78-4d71-9c23-e5bc4112fecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6817","nonce":566077514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6819","nonce":566077514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6823","nonce":566077514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6821","nonce":566077514}]},"public_addr":"172.21.15.187:6817/566077514","cluster_addr":"172.21.15.187:6819/566077514","heartbeat_back_addr":"172.21.15.187:6823/566077514","heartbeat_front_addr":"172.21.15.187:6821/566077514","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:31:38.522095+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:31:56.685873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:15.227272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:31.493769+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:50.070298+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.58:6801/2422532259":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/1616356583":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/2030112153":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/3301163446":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6801/1261811851":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6800/1261811851":"2021-11-21T05:29:42.633679+0000","172.21.15.58:0/1447224424":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6800/324085383":"2021-11-21T05:29:22.266837+0000","172.21.15.58:6801/324085383":"2021-11-21T05:29:22.266837+0000","172.21.15.58:0/574129054":"2021-11-21T05:29:22.266837+0000","172.21.15.58:6800/2422532259":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/2571399354":"2021-11-21T05:29:22.266837+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T05:33:11.148 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:11.146+0000 7f50e1e76700 1 -- 172.21.15.58:0/1195848756 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f50c405c2a0 msgr2=0x7f50c405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:11.148 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:11.146+0000 7f50e1e76700 1 --2- 172.21.15.58:0/1195848756 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f50c405c2a0 0x7f50c405e760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f50dc0f6c10 tx=0x7f50cc00b040).stop 2021-11-20T05:33:11.149 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:11.146+0000 7f50e1e76700 1 -- 172.21.15.58:0/1195848756 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f50dc0f9980 msgr2=0x7f50dc0f5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:11.149 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:11.146+0000 7f50e1e76700 1 --2- 172.21.15.58:0/1195848756 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f50dc0f9980 0x7f50dc0f5300 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f50d0004e30 tx=0x7f50d0009b30).stop 2021-11-20T05:33:11.150 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:11.146+0000 7f50e1e76700 1 -- 172.21.15.58:0/1195848756 shutdown_connections 2021-11-20T05:33:11.150 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:11.146+0000 7f50e1e76700 1 --2- 172.21.15.58:0/1195848756 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f50c405c2a0 0x7f50c405e760 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:11.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:11.146+0000 7f50e1e76700 1 --2- 172.21.15.58:0/1195848756 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50dc060810 0x7f50dc0f6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:11.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:11.146+0000 7f50e1e76700 1 --2- 172.21.15.58:0/1195848756 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f50dc0f9980 0x7f50dc0f5300 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:11.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:11.146+0000 7f50e1e76700 1 -- 172.21.15.58:0/1195848756 >> 172.21.15.58:0/1195848756 conn(0x7f50dc0f0e20 msgr2=0x7f50dc0f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:11.152 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:11.146+0000 7f50e1e76700 1 -- 172.21.15.58:0/1195848756 shutdown_connections 2021-11-20T05:33:11.152 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:11.146+0000 7f50e1e76700 1 -- 172.21.15.58:0/1195848756 wait complete. 2021-11-20T05:33:11.315 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:11 smithi058 conmon[33537]: cephadm 2021-11-20T05:33:09 2021-11-20T05:33:11.316 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:11 smithi058 conmon[33537]: .854725+0000 mgr.smithi058.zagdlk (mgr.14164) 150 : cephadm [INF] Detected new or changed devices on smithi187 2021-11-20T05:33:11.316 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:11 smithi058 conmon[33537]: cluster 2021-11-20T05:33:10.125639+0000 mon.smithi058 (mon.0) 423 : cluster [INF] osd.5 [v2:172.21.15.187:6816/566077514,v1:172.21.15.187:6817/566077514] boot 2021-11-20T05:33:11.316 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:11 smithi058 conmon[33537]: cluster 2021-11-20T05:33:10.125680+0000 mon.smithi058 (mon.0) 424 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2021-11-20T05:33:11.317 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:11 smithi058 conmon[33537]: audit 2021-11-20T05:33:10.125988+0000 mon.smithi058 (mon.0) 425 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T05:33:11.317 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:11 smithi058 conmon[33537]: cluster 2021-11-20T05:33:10.199066+0000 mgr.smithi058.zagdlk (mgr.14164) 151 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:33:11.318 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:11 smithi058 conmon[33537]: audit 2021-11-20T05:33:10.202706+0000 mon.smithi058 (mon.0) 426 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:33:11.318 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:11 smithi058 conmon[33537]: audit 2021-11-20T05:33:10.205728+0000 mon.smithi058 (mon.0) 427 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:33:11.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:11 smithi187 conmon[36358]: cephadm 2021-11-20T05:33:09.854725+0000 mgr.smithi058.zagdlk (mgr.14164) 150 2021-11-20T05:33:11.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:11 smithi187 conmon[36358]: : cephadm [INF] Detected new or changed devices on smithi187 2021-11-20T05:33:11.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:11 smithi187 conmon[36358]: cluster 2021-11-20T05:33:10.125639+0000 mon.smithi058 (mon.0) 423 : cluster [INF] osd.5 [v2:172.21.15.187:6816/566077514,v1:172.21.15.187:6817/566077514] boot 2021-11-20T05:33:11.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:11 smithi187 conmon[36358]: cluster 2021-11-20T05:33:10.125680+0000 mon.smithi058 (mon.0) 424 : cluster 2021-11-20T05:33:11.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:11 smithi187 conmon[36358]: [DBG] osdmap e36: 6 total, 6 up, 6 in 2021-11-20T05:33:11.448 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:11 smithi187 conmon[36358]: audit 2021-11-20T05:33:10.125988+0000 2021-11-20T05:33:11.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:11 smithi187 conmon[36358]: mon.smithi058 (mon.0) 425 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T05:33:11.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:11 smithi187 conmon[36358]: cluster 2021-11-20T05 2021-11-20T05:33:11.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:11 smithi187 conmon[36358]: :33:10.199066+0000 mgr.smithi058.zagdlk (mgr.14164) 151 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T05:33:11.449 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:11 smithi187 conmon[36358]: audit 2021-11-20 2021-11-20T05:33:11.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:11 smithi187 conmon[36358]: T05:33:10.202706+0000 mon.smithi058 (mon.0) 426 : audit [INF] 2021-11-20T05:33:11.450 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:11 smithi187 conmon[36358]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:33:11.451 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:11 smithi187 conmon[36358]: audit 2021-11-20T05: 2021-11-20T05:33:11.451 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:11 smithi187 conmon[36358]: 33:10.205728+0000 mon.smithi058 (mon.0) 427 : 2021-11-20T05:33:11.451 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:11 smithi187 conmon[36358]: audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:33:11.635 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-20T05:32:27.068478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2021-11-20T05:33:11.636 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph osd pool get device_health_metrics pg_num 2021-11-20T05:33:12.101 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:12.395 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:12 smithi058 conmon[33537]: audit 2021-11-20T05:33:11.144908+0000 mon.smithi058 (mon.0) 428 : audit [DBG] from='client.? 172.21.15.58:0/1195848756' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T05:33:12.395 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:12 smithi058 conmon[33537]: cluster 2021-11-20T05:33:11.204441+0000 mon.smithi058 (mon.0) 429 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2021-11-20T05:33:12.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:12 smithi187 conmon[36358]: audit 2021-11-20T05:33:11.144908+0000 mon.smithi058 (mon.0) 428 : audit 2021-11-20T05:33:12.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:12 smithi187 conmon[36358]: [DBG] from='client.? 172.21.15.58:0/1195848756' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T05:33:12.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:12 smithi187 conmon[36358]: cluster 2021-11-20T05:33:11.204441+0000 mon.smithi058 (mon.0) 429 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2021-11-20T05:33:13.344 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:13 smithi058 conmon[33537]: cluster 2021-11-20T05:33:12.199486+0000 mgr.smithi058.zagdlk 2021-11-20T05:33:13.345 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:13 smithi058 conmon[33537]: (mgr.14164) 152 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:13.383 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.383+0000 7fd5ae225700 1 -- 172.21.15.58:0/625046475 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5a80f67e0 msgr2=0x7fd5a80f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:13.384 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.383+0000 7fd5ae225700 1 --2- 172.21.15.58:0/625046475 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5a80f67e0 0x7fd5a80f6c00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fd590004660 tx=0x7fd590009b30).stop 2021-11-20T05:33:13.384 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.383+0000 7fd5ae225700 1 -- 172.21.15.58:0/625046475 shutdown_connections 2021-11-20T05:33:13.384 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.383+0000 7fd5ae225700 1 --2- 172.21.15.58:0/625046475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5a80f7a80 0x7fd5a80f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:13.384 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.383+0000 7fd5ae225700 1 --2- 172.21.15.58:0/625046475 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5a80f67e0 0x7fd5a80f6c00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:13.385 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.383+0000 7fd5ae225700 1 -- 172.21.15.58:0/625046475 >> 172.21.15.58:0/625046475 conn(0x7fd5a80f1f20 msgr2=0x7fd5a80f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:13.385 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.383+0000 7fd5ae225700 1 -- 172.21.15.58:0/625046475 shutdown_connections 2021-11-20T05:33:13.385 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.383+0000 7fd5ae225700 1 -- 172.21.15.58:0/625046475 wait complete. 2021-11-20T05:33:13.385 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.383+0000 7fd5ae225700 1 Processor -- start 2021-11-20T05:33:13.385 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.384+0000 7fd5ae225700 1 -- start start 2021-11-20T05:33:13.386 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.384+0000 7fd5ae225700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5a80f67e0 0x7fd5a806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:13.386 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.384+0000 7fd5ae225700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5a80f7a80 0x7fd5a8069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:13.386 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.384+0000 7fd5ae225700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a806a380 con 0x7fd5a80f67e0 2021-11-20T05:33:13.386 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.384+0000 7fd5ae225700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a806a4c0 con 0x7fd5a80f7a80 2021-11-20T05:33:13.387 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.384+0000 7fd5a77fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5a80f67e0 0x7fd5a806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:13.387 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.384+0000 7fd5a77fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5a80f67e0 0x7fd5a806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40228/0 (socket says 172.21.15.58:40228) 2021-11-20T05:33:13.387 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.384+0000 7fd5a77fe700 1 -- 172.21.15.58:0/3426552501 learned_addr learned my addr 172.21.15.58:0/3426552501 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:13.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.384+0000 7fd5a6ffd700 1 --2- 172.21.15.58:0/3426552501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5a80f7a80 0x7fd5a8069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:13.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.384+0000 7fd5a77fe700 1 -- 172.21.15.58:0/3426552501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5a80f7a80 msgr2=0x7fd5a8069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:13.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.384+0000 7fd5a77fe700 1 --2- 172.21.15.58:0/3426552501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5a80f7a80 0x7fd5a8069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:13.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.384+0000 7fd5a77fe700 1 -- 172.21.15.58:0/3426552501 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd590005040 con 0x7fd5a80f67e0 2021-11-20T05:33:13.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.384+0000 7fd5a6ffd700 1 --2- 172.21.15.58:0/3426552501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5a80f7a80 0x7fd5a8069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:33:13.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.384+0000 7fd5a77fe700 1 --2- 172.21.15.58:0/3426552501 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5a80f67e0 0x7fd5a806b9a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fd590009f50 tx=0x7fd590009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:13.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.385+0000 7fd59ffff700 1 -- 172.21.15.58:0/3426552501 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd590018070 con 0x7fd5a80f67e0 2021-11-20T05:33:13.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.385+0000 7fd59ffff700 1 -- 172.21.15.58:0/3426552501 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd590013410 con 0x7fd5a80f67e0 2021-11-20T05:33:13.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.385+0000 7fd5ae225700 1 -- 172.21.15.58:0/3426552501 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a806a740 con 0x7fd5a80f67e0 2021-11-20T05:33:13.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.385+0000 7fd59ffff700 1 -- 172.21.15.58:0/3426552501 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd59001c620 con 0x7fd5a80f67e0 2021-11-20T05:33:13.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.385+0000 7fd5ae225700 1 -- 172.21.15.58:0/3426552501 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a806aca0 con 0x7fd5a80f67e0 2021-11-20T05:33:13.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.387+0000 7fd59ffff700 1 -- 172.21.15.58:0/3426552501 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd59001d420 con 0x7fd5a80f67e0 2021-11-20T05:33:13.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.387+0000 7fd5ae225700 1 -- 172.21.15.58:0/3426552501 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a8045ad0 con 0x7fd5a80f67e0 2021-11-20T05:33:13.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.387+0000 7fd59ffff700 1 --2- 172.21.15.58:0/3426552501 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd59405c2a0 0x7fd59405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:13.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.387+0000 7fd59ffff700 1 -- 172.21.15.58:0/3426552501 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fd590077940 con 0x7fd5a80f67e0 2021-11-20T05:33:13.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.387+0000 7fd5a6ffd700 1 --2- 172.21.15.58:0/3426552501 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd59405c2a0 0x7fd59405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:13.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.388+0000 7fd5a6ffd700 1 --2- 172.21.15.58:0/3426552501 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd59405c2a0 0x7fd59405e760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fd5980041d0 tx=0x7fd598009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:13.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.392+0000 7fd59ffff700 1 -- 172.21.15.58:0/3426552501 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd59004a9c0 con 0x7fd5a80f67e0 2021-11-20T05:33:13.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:13 smithi187 conmon[36358]: cluster 2021-11-20T05:33:12.199486+0000 mgr.smithi058.zagdlk (mgr.14164) 152 : cluster [DBG] 2021-11-20T05:33:13.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:13 smithi187 conmon[36358]: pgmap v122: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:13.555 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.555+0000 7fd5ae225700 1 -- 172.21.15.58:0/3426552501 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fd5a80ff0b0 con 0x7fd5a80f67e0 2021-11-20T05:33:13.556 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.555+0000 7fd59ffff700 1 -- 172.21.15.58:0/3426552501 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v37) v1 ==== 110+0+10 (secure 0 0 0) 0x7fd59004dfe0 con 0x7fd5a80f67e0 2021-11-20T05:33:13.556 INFO:teuthology.orchestra.run.smithi058.stdout:pg_num: 1 2021-11-20T05:33:13.557 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.557+0000 7fd5ae225700 1 -- 172.21.15.58:0/3426552501 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd59405c2a0 msgr2=0x7fd59405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:13.557 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.557+0000 7fd5ae225700 1 --2- 172.21.15.58:0/3426552501 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd59405c2a0 0x7fd59405e760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fd5980041d0 tx=0x7fd598009040).stop 2021-11-20T05:33:13.558 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.557+0000 7fd5ae225700 1 -- 172.21.15.58:0/3426552501 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5a80f67e0 msgr2=0x7fd5a806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:13.558 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.557+0000 7fd5ae225700 1 --2- 172.21.15.58:0/3426552501 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5a80f67e0 0x7fd5a806b9a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fd590009f50 tx=0x7fd590009b30).stop 2021-11-20T05:33:13.558 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.557+0000 7fd5ae225700 1 -- 172.21.15.58:0/3426552501 shutdown_connections 2021-11-20T05:33:13.558 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.557+0000 7fd5ae225700 1 --2- 172.21.15.58:0/3426552501 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd59405c2a0 0x7fd59405e760 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:13.559 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.557+0000 7fd5ae225700 1 --2- 172.21.15.58:0/3426552501 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5a80f67e0 0x7fd5a806b9a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:13.559 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.557+0000 7fd5ae225700 1 --2- 172.21.15.58:0/3426552501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5a80f7a80 0x7fd5a8069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:13.559 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.557+0000 7fd5ae225700 1 -- 172.21.15.58:0/3426552501 >> 172.21.15.58:0/3426552501 conn(0x7fd5a80f1f20 msgr2=0x7fd5a80fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:13.560 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.558+0000 7fd5ae225700 1 -- 172.21.15.58:0/3426552501 shutdown_connections 2021-11-20T05:33:13.560 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:13.558+0000 7fd5ae225700 1 -- 172.21.15.58:0/3426552501 wait complete. 2021-11-20T05:33:14.153 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-11-20T05:33:14.154 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2021-11-20T05:33:14.155 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph mgr dump --format=json 2021-11-20T05:33:14.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:14 smithi187 conmon[36358]: audit 2021-11-20T05:33:13.556146+0000 mon.smithi058 (mon.0) 2021-11-20T05:33:14.447 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:14 smithi187 conmon[36358]: 430 : audit [DBG] from='client.? 172.21.15.58:0/3426552501' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T05:33:14.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:14 smithi058 conmon[33537]: audit 2021-11-20T05:33:13.556146+0000 mon.smithi058 (mon 2021-11-20T05:33:14.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:14 smithi058 conmon[33537]: .0) 430 : audit [DBG] from='client.? 172.21.15.58:0/3426552501' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T05:33:14.659 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:15.441 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:15 smithi058 conmon[33537]: cluster 2021-11-20T05:33:14.199803+0000 mgr.smithi058.zagdlk (mgr.14164) 153 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:15.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:15 smithi187 conmon[36358]: cluster 2021-11-20T05:33:14.199803+0000 mgr.smithi058.zagdlk (mgr.14164) 153 : cluster 2021-11-20T05:33:15.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:15 smithi187 conmon[36358]: [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:16.011 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.011+0000 7f773144e700 1 -- 172.21.15.58:0/4046907085 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f772c0f6960 msgr2=0x7f772c0f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:16.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.011+0000 7f773144e700 1 --2- 172.21.15.58:0/4046907085 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f772c0f6960 0x7f772c0f6dc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f77200046e0 tx=0x7f7720009b30).stop 2021-11-20T05:33:16.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.011+0000 7f773144e700 1 -- 172.21.15.58:0/4046907085 shutdown_connections 2021-11-20T05:33:16.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.011+0000 7f773144e700 1 --2- 172.21.15.58:0/4046907085 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f772c0f6960 0x7f772c0f6dc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:16.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.011+0000 7f773144e700 1 --2- 172.21.15.58:0/4046907085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f772c0f56c0 0x7f772c0f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:33:16.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.011+0000 7f773144e700 1 -- 172.21.15.58:0/4046907085 >> 172.21.15.58:0/4046907085 conn(0x7f772c0f0e20 msgr2=0x7f772c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:16.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.011+0000 7f773144e700 1 -- 172.21.15.58:0/4046907085 shutdown_connections 2021-11-20T05:33:16.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.011+0000 7f773144e700 1 -- 172.21.15.58:0/4046907085 wait complete. 2021-11-20T05:33:16.014 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.012+0000 7f773144e700 1 Processor -- start 2021-11-20T05:33:16.014 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.012+0000 7f773144e700 1 -- start start 2021-11-20T05:33:16.016 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.012+0000 7f773144e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f772c0f56c0 0x7f772c104ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:16.017 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.012+0000 7f773144e700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f772c0f6960 0x7f772c1053e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:16.017 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.013+0000 7f773144e700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772c105a20 con 0x7f772c0f6960 2021-11-20T05:33:16.017 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.013+0000 7f773144e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772c1062d0 con 0x7f772c0f56c0 2021-11-20T05:33:16.017 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.013+0000 7f772a7fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f772c0f6960 0x7f772c1053e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:16.017 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.013+0000 7f772a7fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f772c0f6960 0x7f772c1053e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40240/0 (socket says 172.21.15.58:40240) 2021-11-20T05:33:16.018 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.013+0000 7f772a7fc700 1 -- 172.21.15.58:0/3410143381 learned_addr learned my addr 172.21.15.58:0/3410143381 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:16.018 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.013+0000 7f772affd700 1 --2- 172.21.15.58:0/3410143381 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f772c0f56c0 0x7f772c104ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:16.018 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.013+0000 7f772a7fc700 1 -- 172.21.15.58:0/3410143381 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f772c0f56c0 msgr2=0x7f772c104ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:16.018 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.013+0000 7f772a7fc700 1 --2- 172.21.15.58:0/3410143381 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f772c0f56c0 0x7f772c104ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:16.019 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.013+0000 7f772a7fc700 1 -- 172.21.15.58:0/3410143381 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7720005040 con 0x7f772c0f6960 2021-11-20T05:33:16.019 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.013+0000 7f772a7fc700 1 --2- 172.21.15.58:0/3410143381 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f772c0f6960 0x7f772c1053e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f7720014040 tx=0x7f7720009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:16.019 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.013+0000 7f77137fe700 1 -- 172.21.15.58:0/3410143381 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7720007cf0 con 0x7f772c0f6960 2021-11-20T05:33:16.019 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.013+0000 7f77137fe700 1 -- 172.21.15.58:0/3410143381 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7720007e50 con 0x7f772c0f6960 2021-11-20T05:33:16.020 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.013+0000 7f77137fe700 1 -- 172.21.15.58:0/3410143381 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f772001b650 con 0x7f772c0f6960 2021-11-20T05:33:16.020 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.013+0000 7f773144e700 1 -- 172.21.15.58:0/3410143381 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f772c106550 con 0x7f772c0f6960 2021-11-20T05:33:16.020 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.013+0000 7f773144e700 1 -- 172.21.15.58:0/3410143381 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f772c106a60 con 0x7f772c0f6960 2021-11-20T05:33:16.020 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.014+0000 7f773144e700 1 -- 172.21.15.58:0/3410143381 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f772c059100 con 0x7f772c0f6960 2021-11-20T05:33:16.021 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.015+0000 7f77137fe700 1 -- 172.21.15.58:0/3410143381 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f772002b070 con 0x7f772c0f6960 2021-11-20T05:33:16.021 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.015+0000 7f77137fe700 1 --2- 172.21.15.58:0/3410143381 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f771405c2a0 0x7f771405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:16.021 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.015+0000 7f77137fe700 1 -- 172.21.15.58:0/3410143381 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7720079c10 con 0x7f772c0f6960 2021-11-20T05:33:16.021 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.015+0000 7f772affd700 1 --2- 172.21.15.58:0/3410143381 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f771405c2a0 0x7f771405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:16.022 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.016+0000 7f772affd700 1 --2- 172.21.15.58:0/3410143381 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f771405c2a0 0x7f771405e760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f771c006fd0 tx=0x7f771c008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:16.022 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.019+0000 7f77137fe700 1 -- 172.21.15.58:0/3410143381 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f772004cc10 con 0x7f772c0f6960 2021-11-20T05:33:16.222 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.223+0000 7f773144e700 1 -- 172.21.15.58:0/3410143381 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f772c0fce20 con 0x7f772c0f6960 2021-11-20T05:33:16.228 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.228+0000 7f77137fe700 1 -- 172.21.15.58:0/3410143381 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v18) v1 ==== 74+0+143497 (secure 0 0 0) 0x7f772001f020 con 0x7f772c0f6960 2021-11-20T05:33:16.231 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:33:16.233 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.231+0000 7f773144e700 1 -- 172.21.15.58:0/3410143381 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f771405c2a0 msgr2=0x7f771405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:16.233 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.231+0000 7f773144e700 1 --2- 172.21.15.58:0/3410143381 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f771405c2a0 0x7f771405e760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f771c006fd0 tx=0x7f771c008040).stop 2021-11-20T05:33:16.233 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.231+0000 7f773144e700 1 -- 172.21.15.58:0/3410143381 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f772c0f6960 msgr2=0x7f772c1053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:16.235 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.231+0000 7f773144e700 1 --2- 172.21.15.58:0/3410143381 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f772c0f6960 0x7f772c1053e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f7720014040 tx=0x7f7720009b30).stop 2021-11-20T05:33:16.235 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.231+0000 7f773144e700 1 -- 172.21.15.58:0/3410143381 shutdown_connections 2021-11-20T05:33:16.236 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.231+0000 7f773144e700 1 --2- 172.21.15.58:0/3410143381 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f771405c2a0 0x7f771405e760 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:16.236 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.231+0000 7f773144e700 1 --2- 172.21.15.58:0/3410143381 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f772c0f56c0 0x7f772c104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:16.236 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.231+0000 7f773144e700 1 --2- 172.21.15.58:0/3410143381 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f772c0f6960 0x7f772c1053e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:16.237 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.231+0000 7f773144e700 1 -- 172.21.15.58:0/3410143381 >> 172.21.15.58:0/3410143381 conn(0x7f772c0f0e20 msgr2=0x7f772c0fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:16.237 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.232+0000 7f773144e700 1 -- 172.21.15.58:0/3410143381 shutdown_connections 2021-11-20T05:33:16.238 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:16.232+0000 7f773144e700 1 -- 172.21.15.58:0/3410143381 wait complete. 2021-11-20T05:33:16.574 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":18,"active_gid":14164,"active_name":"smithi058.zagdlk","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6800","nonce":2097301011},{"type":"v1","addr":"172.21.15.58:6801","nonce":2097301011}]},"active_addr":"172.21.15.58:6801/2097301011","active_change":"2021-11-20T05:29:59.312002+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":14200,"name":"smithi187.omkctn","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://smithi058.front.sepia.ceph.com:8443/","prometheus":"http://smithi058.front.sepia.ceph.com:9283/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":4,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.58:0","nonce":497157330}]},{"addrvec":[{"type":"v2","addr":"172.21.15.58:0","nonce":34204117}]}]}} 2021-11-20T05:33:16.579 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2021-11-20T05:33:16.579 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2021-11-20T05:33:16.580 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph osd dump --format=json 2021-11-20T05:33:17.023 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:17.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:17 smithi058 conmon[33537]: cluster 2021-11-20T05:33:16.200258+0000 mgr.smithi058.zagdlk (mgr.14164) 154 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:17.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:17 smithi058 conmon[33537]: audit 2021-11-20T05:33:16.224481+0000 mon.smithi058 (mon.0) 431 : audit [DBG] from='client.? 172.21.15.58:0/3410143381' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-11-20T05:33:17.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:17 smithi058 conmon[33537]: audit 2021-11-20T05:33:16.259485+0000 mon.smithi058 (mon.0) 432 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:33:17.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:17 smithi058 conmon[33537]: audit 2021-11-20T05:33:16.263664+0000 mon.smithi058 (mon.0) 433 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:33:17.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:17 smithi187 conmon[36358]: cluster 2021-11-20T05:33:16.200258+0000 mgr.smithi058.zagdlk (mgr.14164) 154 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:17.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:17 smithi187 conmon[36358]: audit 2021-11-20T05:33:16.224481+0000 mon.smithi058 (mon.0) 431 : audit [DBG] from='client.? 172.21.15.58:0/3410143381' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-11-20T05:33:17.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:17 smithi187 conmon[36358]: audit 2021-11-20T05:33:16.259485+0000 mon.smithi058 (mon.0) 2021-11-20T05:33:17.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:17 smithi187 conmon[36358]: 432 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:33:17.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:17 smithi187 conmon[36358]: audit 2021-11-20 2021-11-20T05:33:17.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:17 smithi187 conmon[36358]: T05:33:16.263664+0000 mon.smithi058 (mon.0) 433 : audit 2021-11-20T05:33:17.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:17 smithi187 conmon[36358]: [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:33:18.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.094+0000 7fb9f19a7700 1 -- 172.21.15.58:0/2094336618 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb9ec0f6cc0 msgr2=0x7fb9ec0f70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:18.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.094+0000 7fb9f19a7700 1 --2- 172.21.15.58:0/2094336618 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb9ec0f6cc0 0x7fb9ec0f70e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fb9dc004660 tx=0x7fb9dc009b30).stop 2021-11-20T05:33:18.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.094+0000 7fb9f19a7700 1 -- 172.21.15.58:0/2094336618 shutdown_connections 2021-11-20T05:33:18.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.094+0000 7fb9f19a7700 1 --2- 172.21.15.58:0/2094336618 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9ec0f7660 0x7fb9ec0f5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:18.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.094+0000 7fb9f19a7700 1 --2- 172.21.15.58:0/2094336618 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb9ec0f6cc0 0x7fb9ec0f70e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:18.097 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.094+0000 7fb9f19a7700 1 -- 172.21.15.58:0/2094336618 >> 172.21.15.58:0/2094336618 conn(0x7fb9ec0f0590 msgr2=0x7fb9ec0f29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:18.097 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.094+0000 7fb9f19a7700 1 -- 172.21.15.58:0/2094336618 shutdown_connections 2021-11-20T05:33:18.097 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.094+0000 7fb9f19a7700 1 -- 172.21.15.58:0/2094336618 wait complete. 2021-11-20T05:33:18.097 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.095+0000 7fb9f19a7700 1 Processor -- start 2021-11-20T05:33:18.098 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.095+0000 7fb9f19a7700 1 -- start start 2021-11-20T05:33:18.098 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.095+0000 7fb9f19a7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9ec0f6cc0 0x7fb9ec107080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:18.098 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.095+0000 7fb9f19a7700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb9ec0f7660 0x7fb9ec107570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:18.098 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.095+0000 7fb9f19a7700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec107b20 con 0x7fb9ec0f7660 2021-11-20T05:33:18.099 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.095+0000 7fb9f19a7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec107c60 con 0x7fb9ec0f6cc0 2021-11-20T05:33:18.099 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.095+0000 7fb9ea7fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb9ec0f7660 0x7fb9ec107570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:18.099 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.095+0000 7fb9ea7fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb9ec0f7660 0x7fb9ec107570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40250/0 (socket says 172.21.15.58:40250) 2021-11-20T05:33:18.099 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.095+0000 7fb9ea7fc700 1 -- 172.21.15.58:0/1555345604 learned_addr learned my addr 172.21.15.58:0/1555345604 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:18.099 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.095+0000 7fb9eaffd700 1 --2- 172.21.15.58:0/1555345604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9ec0f6cc0 0x7fb9ec107080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:18.100 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.096+0000 7fb9ea7fc700 1 -- 172.21.15.58:0/1555345604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9ec0f6cc0 msgr2=0x7fb9ec107080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:18.100 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.096+0000 7fb9ea7fc700 1 --2- 172.21.15.58:0/1555345604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9ec0f6cc0 0x7fb9ec107080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:18.100 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.096+0000 7fb9ea7fc700 1 -- 172.21.15.58:0/1555345604 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc005040 con 0x7fb9ec0f7660 2021-11-20T05:33:18.100 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.096+0000 7fb9ea7fc700 1 --2- 172.21.15.58:0/1555345604 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb9ec0f7660 0x7fb9ec107570 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fb9e00026d0 tx=0x7fb9e0009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:18.100 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.096+0000 7fb9d37fe700 1 -- 172.21.15.58:0/1555345604 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9e00079c0 con 0x7fb9ec0f7660 2021-11-20T05:33:18.101 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.096+0000 7fb9f19a7700 1 -- 172.21.15.58:0/1555345604 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9ec108600 con 0x7fb9ec0f7660 2021-11-20T05:33:18.101 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.096+0000 7fb9d37fe700 1 -- 172.21.15.58:0/1555345604 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9e000f040 con 0x7fb9ec0f7660 2021-11-20T05:33:18.101 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.096+0000 7fb9d37fe700 1 -- 172.21.15.58:0/1555345604 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9e00148b0 con 0x7fb9ec0f7660 2021-11-20T05:33:18.101 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.096+0000 7fb9f19a7700 1 -- 172.21.15.58:0/1555345604 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9ec108bc0 con 0x7fb9ec0f7660 2021-11-20T05:33:18.103 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.097+0000 7fb9f19a7700 1 -- 172.21.15.58:0/1555345604 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9ec045ad0 con 0x7fb9ec0f7660 2021-11-20T05:33:18.103 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.097+0000 7fb9d37fe700 1 -- 172.21.15.58:0/1555345604 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb9e0007b20 con 0x7fb9ec0f7660 2021-11-20T05:33:18.103 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.098+0000 7fb9d37fe700 1 --2- 172.21.15.58:0/1555345604 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb9d405c2a0 0x7fb9d405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:18.103 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.098+0000 7fb9d37fe700 1 -- 172.21.15.58:0/1555345604 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fb9e0075b00 con 0x7fb9ec0f7660 2021-11-20T05:33:18.103 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.098+0000 7fb9eaffd700 1 --2- 172.21.15.58:0/1555345604 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb9d405c2a0 0x7fb9d405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:18.104 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.098+0000 7fb9eaffd700 1 --2- 172.21.15.58:0/1555345604 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb9d405c2a0 0x7fb9d405e760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fb9dc009f50 tx=0x7fb9dc009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:18.104 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.102+0000 7fb9d37fe700 1 -- 172.21.15.58:0/1555345604 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9e0048b00 con 0x7fb9ec0f7660 2021-11-20T05:33:18.263 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.263+0000 7fb9f19a7700 1 -- 172.21.15.58:0/1555345604 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9ec10d6a0 con 0x7fb9ec0f7660 2021-11-20T05:33:18.264 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.264+0000 7fb9d37fe700 1 -- 172.21.15.58:0/1555345604 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+10909 (secure 0 0 0) 0x7fb9e004c120 con 0x7fb9ec0f7660 2021-11-20T05:33:18.265 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:33:18.265 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":37,"fsid":"b33f8c4a-49c2-11ec-8c2c-001a4aab830c","created":"2021-11-20T05:28:54.787342+0000","modified":"2021-11-20T05:33:11.201018+0000","last_up_change":"2021-11-20T05:33:10.122001+0000","last_in_change":"2021-11-20T05:32:53.577246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T05:32:27.068478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"e2f5103a-e0d2-4bc3-88e1-f28420c98071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6802","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6803","nonce":2375873844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6804","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6805","nonce":2375873844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6808","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6809","nonce":2375873844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6806","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6807","nonce":2375873844}]},"public_addr":"172.21.15.58:6803/2375873844","cluster_addr":"172.21.15.58:6805/2375873844","heartbeat_back_addr":"172.21.15.58:6809/2375873844","heartbeat_front_addr":"172.21.15.58:6807/2375873844","state":["exists","up"]},{"osd":1,"uuid":"e0243e40-c76e-4c68-8ac0-8f555035e948","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6810","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6811","nonce":3875723869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6812","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6813","nonce":3875723869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6816","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6817","nonce":3875723869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6814","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6815","nonce":3875723869}]},"public_addr":"172.21.15.58:6811/3875723869","cluster_addr":"172.21.15.58:6813/3875723869","heartbeat_back_addr":"172.21.15.58:6817/3875723869","heartbeat_front_addr":"172.21.15.58:6815/3875723869","state":["exists","up"]},{"osd":2,"uuid":"9b35ec54-f60f-49c5-ab1b-d2c5b5eb72e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6818","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6819","nonce":745015076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6820","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6821","nonce":745015076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6824","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6825","nonce":745015076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6822","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6823","nonce":745015076}]},"public_addr":"172.21.15.58:6819/745015076","cluster_addr":"172.21.15.58:6821/745015076","heartbeat_back_addr":"172.21.15.58:6825/745015076","heartbeat_front_addr":"172.21.15.58:6823/745015076","state":["exists","up"]},{"osd":3,"uuid":"8390f146-b7e1-4039-83d3-7ed1c832dc8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6801","nonce":1941886381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6803","nonce":1941886381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6807","nonce":1941886381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6805","nonce":1941886381}]},"public_addr":"172.21.15.187:6801/1941886381","cluster_addr":"172.21.15.187:6803/1941886381","heartbeat_back_addr":"172.21.15.187:6807/1941886381","heartbeat_front_addr":"172.21.15.187:6805/1941886381","state":["exists","up"]},{"osd":4,"uuid":"89dc0c2f-c52c-48ca-b3ab-792e299d494d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6809","nonce":3483552299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6811","nonce":3483552299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6815","nonce":3483552299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6813","nonce":3483552299}]},"public_addr":"172.21.15.187:6809/3483552299","cluster_addr":"172.21.15.187:6811/3483552299","heartbeat_back_addr":"172.21.15.187:6815/3483552299","heartbeat_front_addr":"172.21.15.187:6813/3483552299","state":["exists","up"]},{"osd":5,"uuid":"8190ded3-7b78-4d71-9c23-e5bc4112fecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6817","nonce":566077514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6819","nonce":566077514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6823","nonce":566077514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6821","nonce":566077514}]},"public_addr":"172.21.15.187:6817/566077514","cluster_addr":"172.21.15.187:6819/566077514","heartbeat_back_addr":"172.21.15.187:6823/566077514","heartbeat_front_addr":"172.21.15.187:6821/566077514","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:31:38.522095+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:31:56.685873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:15.227272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:31.493769+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:50.070298+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:33:08.140441+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.58:6801/2422532259":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/1616356583":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/2030112153":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/3301163446":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6801/1261811851":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6800/1261811851":"2021-11-21T05:29:42.633679+0000","172.21.15.58:0/1447224424":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6800/324085383":"2021-11-21T05:29:22.266837+0000","172.21.15.58:6801/324085383":"2021-11-21T05:29:22.266837+0000","172.21.15.58:0/574129054":"2021-11-21T05:29:22.266837+0000","172.21.15.58:6800/2422532259":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/2571399354":"2021-11-21T05:29:22.266837+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T05:33:18.266 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.265+0000 7fb9f19a7700 1 -- 172.21.15.58:0/1555345604 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb9d405c2a0 msgr2=0x7fb9d405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:18.266 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.265+0000 7fb9f19a7700 1 --2- 172.21.15.58:0/1555345604 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb9d405c2a0 0x7fb9d405e760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fb9dc009f50 tx=0x7fb9dc009b30).stop 2021-11-20T05:33:18.267 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.265+0000 7fb9f19a7700 1 -- 172.21.15.58:0/1555345604 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb9ec0f7660 msgr2=0x7fb9ec107570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:18.267 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.265+0000 7fb9f19a7700 1 --2- 172.21.15.58:0/1555345604 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb9ec0f7660 0x7fb9ec107570 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fb9e00026d0 tx=0x7fb9e0009de0).stop 2021-11-20T05:33:18.267 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.266+0000 7fb9f19a7700 1 -- 172.21.15.58:0/1555345604 shutdown_connections 2021-11-20T05:33:18.267 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.266+0000 7fb9f19a7700 1 --2- 172.21.15.58:0/1555345604 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb9d405c2a0 0x7fb9d405e760 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:18.268 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.266+0000 7fb9f19a7700 1 --2- 172.21.15.58:0/1555345604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9ec0f6cc0 0x7fb9ec107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:18.268 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.266+0000 7fb9f19a7700 1 --2- 172.21.15.58:0/1555345604 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb9ec0f7660 0x7fb9ec107570 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:18.268 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.266+0000 7fb9f19a7700 1 -- 172.21.15.58:0/1555345604 >> 172.21.15.58:0/1555345604 conn(0x7fb9ec0f0590 msgr2=0x7fb9ec0f5170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:18.269 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.266+0000 7fb9f19a7700 1 -- 172.21.15.58:0/1555345604 shutdown_connections 2021-11-20T05:33:18.269 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:18.266+0000 7fb9f19a7700 1 -- 172.21.15.58:0/1555345604 wait complete. 2021-11-20T05:33:18.724 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2021-11-20T05:33:18.724 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph osd dump --format=json 2021-11-20T05:33:19.168 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:19.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:19 smithi058 conmon[33537]: cluster 2021-11-20T05:33:18.200546+0000 mgr.smithi058.zagdlk (mgr.14164) 155 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:19.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:19 smithi058 conmon[33537]: audit 2021-11-20T05:33:18.264617+0000 mon.smithi058 (mon.0) 434 : audit [DBG] from='client.? 172.21.15.58:0/1555345604' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T05:33:19.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:19 smithi187 conmon[36358]: cluster 2021-11-20T05:33:18.200546+0000 mgr.smithi058.zagdlk (mgr.14164) 155 : cluster [DBG] 2021-11-20T05:33:19.796 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:19 smithi187 conmon[36358]: pgmap v125: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:19.797 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:19 smithi187 conmon[36358]: audit 2021-11-20T05:33:18.264617+0000 mon.smithi058 (mon.0) 434 : audit [DBG] from='client.? 172.21.15.58:0/1555345604' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T05:33:20.315 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.315+0000 7fb4cdee8700 1 -- 172.21.15.58:0/2221798184 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb4c80f4aa0 msgr2=0x7fb4c80f4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:20.316 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.315+0000 7fb4cdee8700 1 --2- 172.21.15.58:0/2221798184 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb4c80f4aa0 0x7fb4c80f4ec0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fb4b8004660 tx=0x7fb4b8009b30).stop 2021-11-20T05:33:20.316 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.315+0000 7fb4cdee8700 1 -- 172.21.15.58:0/2221798184 shutdown_connections 2021-11-20T05:33:20.316 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.315+0000 7fb4cdee8700 1 --2- 172.21.15.58:0/2221798184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4c80f5440 0x7fb4c80f7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:20.316 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.315+0000 7fb4cdee8700 1 --2- 172.21.15.58:0/2221798184 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb4c80f4aa0 0x7fb4c80f4ec0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:20.317 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.315+0000 7fb4cdee8700 1 -- 172.21.15.58:0/2221798184 >> 172.21.15.58:0/2221798184 conn(0x7fb4c80f05e0 msgr2=0x7fb4c80f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:20.317 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.315+0000 7fb4cdee8700 1 -- 172.21.15.58:0/2221798184 shutdown_connections 2021-11-20T05:33:20.317 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.315+0000 7fb4cdee8700 1 -- 172.21.15.58:0/2221798184 wait complete. 2021-11-20T05:33:20.317 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.316+0000 7fb4cdee8700 1 Processor -- start 2021-11-20T05:33:20.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.316+0000 7fb4cdee8700 1 -- start start 2021-11-20T05:33:20.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.316+0000 7fb4cdee8700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb4c80f4aa0 0x7fb4c8107100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:20.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.316+0000 7fb4cdee8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4c80f5440 0x7fb4c81075f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:20.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.316+0000 7fb4cdee8700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c8107ba0 con 0x7fb4c80f4aa0 2021-11-20T05:33:20.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.316+0000 7fb4cdee8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c8107ce0 con 0x7fb4c80f5440 2021-11-20T05:33:20.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.316+0000 7fb4c77fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb4c80f4aa0 0x7fb4c8107100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:20.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.316+0000 7fb4c77fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb4c80f4aa0 0x7fb4c8107100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40256/0 (socket says 172.21.15.58:40256) 2021-11-20T05:33:20.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.316+0000 7fb4c77fe700 1 -- 172.21.15.58:0/300707387 learned_addr learned my addr 172.21.15.58:0/300707387 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:20.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.316+0000 7fb4c6ffd700 1 --2- 172.21.15.58:0/300707387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4c80f5440 0x7fb4c81075f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:20.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.317+0000 7fb4c77fe700 1 -- 172.21.15.58:0/300707387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4c80f5440 msgr2=0x7fb4c81075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:20.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.317+0000 7fb4c77fe700 1 --2- 172.21.15.58:0/300707387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4c80f5440 0x7fb4c81075f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:20.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.317+0000 7fb4c77fe700 1 -- 172.21.15.58:0/300707387 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b8005040 con 0x7fb4c80f4aa0 2021-11-20T05:33:20.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.317+0000 7fb4c6ffd700 1 --2- 172.21.15.58:0/300707387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4c80f5440 0x7fb4c81075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:33:20.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.317+0000 7fb4c77fe700 1 --2- 172.21.15.58:0/300707387 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb4c80f4aa0 0x7fb4c8107100 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb4b8004e30 tx=0x7fb4b8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:20.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.317+0000 7fb4bffff700 1 -- 172.21.15.58:0/300707387 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4b8018070 con 0x7fb4c80f4aa0 2021-11-20T05:33:20.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.317+0000 7fb4bffff700 1 -- 172.21.15.58:0/300707387 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4b8013410 con 0x7fb4c80f4aa0 2021-11-20T05:33:20.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.317+0000 7fb4cdee8700 1 -- 172.21.15.58:0/300707387 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c8108620 con 0x7fb4c80f4aa0 2021-11-20T05:33:20.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.317+0000 7fb4bffff700 1 -- 172.21.15.58:0/300707387 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4b801c3c0 con 0x7fb4c80f4aa0 2021-11-20T05:33:20.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.317+0000 7fb4cdee8700 1 -- 172.21.15.58:0/300707387 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c8108b80 con 0x7fb4c80f4aa0 2021-11-20T05:33:20.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.318+0000 7fb4cdee8700 1 -- 172.21.15.58:0/300707387 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c8045ad0 con 0x7fb4c80f4aa0 2021-11-20T05:33:20.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.318+0000 7fb4bffff700 1 -- 172.21.15.58:0/300707387 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb4b801d420 con 0x7fb4c80f4aa0 2021-11-20T05:33:20.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.319+0000 7fb4bffff700 1 --2- 172.21.15.58:0/300707387 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb4b405c2a0 0x7fb4b405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:20.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.319+0000 7fb4bffff700 1 -- 172.21.15.58:0/300707387 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fb4b8077820 con 0x7fb4c80f4aa0 2021-11-20T05:33:20.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.319+0000 7fb4c6ffd700 1 --2- 172.21.15.58:0/300707387 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb4b405c2a0 0x7fb4b405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:20.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.319+0000 7fb4c6ffd700 1 --2- 172.21.15.58:0/300707387 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb4b405c2a0 0x7fb4b405e760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fb4c80f5880 tx=0x7fb4b0009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:20.326 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.323+0000 7fb4bffff700 1 -- 172.21.15.58:0/300707387 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4b804a820 con 0x7fb4c80f4aa0 2021-11-20T05:33:20.481 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.482+0000 7fb4cdee8700 1 -- 172.21.15.58:0/300707387 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4c8027a00 con 0x7fb4c80f4aa0 2021-11-20T05:33:20.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.482+0000 7fb4bffff700 1 -- 172.21.15.58:0/300707387 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+10909 (secure 0 0 0) 0x7fb4b804de40 con 0x7fb4c80f4aa0 2021-11-20T05:33:20.484 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:33:20.485 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":37,"fsid":"b33f8c4a-49c2-11ec-8c2c-001a4aab830c","created":"2021-11-20T05:28:54.787342+0000","modified":"2021-11-20T05:33:11.201018+0000","last_up_change":"2021-11-20T05:33:10.122001+0000","last_in_change":"2021-11-20T05:32:53.577246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T05:32:27.068478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"e2f5103a-e0d2-4bc3-88e1-f28420c98071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6802","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6803","nonce":2375873844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6804","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6805","nonce":2375873844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6808","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6809","nonce":2375873844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6806","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6807","nonce":2375873844}]},"public_addr":"172.21.15.58:6803/2375873844","cluster_addr":"172.21.15.58:6805/2375873844","heartbeat_back_addr":"172.21.15.58:6809/2375873844","heartbeat_front_addr":"172.21.15.58:6807/2375873844","state":["exists","up"]},{"osd":1,"uuid":"e0243e40-c76e-4c68-8ac0-8f555035e948","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6810","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6811","nonce":3875723869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6812","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6813","nonce":3875723869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6816","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6817","nonce":3875723869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6814","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6815","nonce":3875723869}]},"public_addr":"172.21.15.58:6811/3875723869","cluster_addr":"172.21.15.58:6813/3875723869","heartbeat_back_addr":"172.21.15.58:6817/3875723869","heartbeat_front_addr":"172.21.15.58:6815/3875723869","state":["exists","up"]},{"osd":2,"uuid":"9b35ec54-f60f-49c5-ab1b-d2c5b5eb72e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6818","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6819","nonce":745015076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6820","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6821","nonce":745015076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6824","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6825","nonce":745015076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6822","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6823","nonce":745015076}]},"public_addr":"172.21.15.58:6819/745015076","cluster_addr":"172.21.15.58:6821/745015076","heartbeat_back_addr":"172.21.15.58:6825/745015076","heartbeat_front_addr":"172.21.15.58:6823/745015076","state":["exists","up"]},{"osd":3,"uuid":"8390f146-b7e1-4039-83d3-7ed1c832dc8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6801","nonce":1941886381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6803","nonce":1941886381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6807","nonce":1941886381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6805","nonce":1941886381}]},"public_addr":"172.21.15.187:6801/1941886381","cluster_addr":"172.21.15.187:6803/1941886381","heartbeat_back_addr":"172.21.15.187:6807/1941886381","heartbeat_front_addr":"172.21.15.187:6805/1941886381","state":["exists","up"]},{"osd":4,"uuid":"89dc0c2f-c52c-48ca-b3ab-792e299d494d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6809","nonce":3483552299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6811","nonce":3483552299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6815","nonce":3483552299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6813","nonce":3483552299}]},"public_addr":"172.21.15.187:6809/3483552299","cluster_addr":"172.21.15.187:6811/3483552299","heartbeat_back_addr":"172.21.15.187:6815/3483552299","heartbeat_front_addr":"172.21.15.187:6813/3483552299","state":["exists","up"]},{"osd":5,"uuid":"8190ded3-7b78-4d71-9c23-e5bc4112fecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6817","nonce":566077514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6819","nonce":566077514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6823","nonce":566077514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6821","nonce":566077514}]},"public_addr":"172.21.15.187:6817/566077514","cluster_addr":"172.21.15.187:6819/566077514","heartbeat_back_addr":"172.21.15.187:6823/566077514","heartbeat_front_addr":"172.21.15.187:6821/566077514","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:31:38.522095+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:31:56.685873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:15.227272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:31.493769+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:50.070298+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:33:08.140441+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.58:6801/2422532259":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/1616356583":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/2030112153":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/3301163446":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6801/1261811851":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6800/1261811851":"2021-11-21T05:29:42.633679+0000","172.21.15.58:0/1447224424":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6800/324085383":"2021-11-21T05:29:22.266837+0000","172.21.15.58:6801/324085383":"2021-11-21T05:29:22.266837+0000","172.21.15.58:0/574129054":"2021-11-21T05:29:22.266837+0000","172.21.15.58:6800/2422532259":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/2571399354":"2021-11-21T05:29:22.266837+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T05:33:20.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.484+0000 7fb4cdee8700 1 -- 172.21.15.58:0/300707387 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb4b405c2a0 msgr2=0x7fb4b405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:20.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.484+0000 7fb4cdee8700 1 --2- 172.21.15.58:0/300707387 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb4b405c2a0 0x7fb4b405e760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fb4c80f5880 tx=0x7fb4b0009be0).stop 2021-11-20T05:33:20.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.484+0000 7fb4cdee8700 1 -- 172.21.15.58:0/300707387 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb4c80f4aa0 msgr2=0x7fb4c8107100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:20.488 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.484+0000 7fb4cdee8700 1 --2- 172.21.15.58:0/300707387 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb4c80f4aa0 0x7fb4c8107100 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb4b8004e30 tx=0x7fb4b8009b30).stop 2021-11-20T05:33:20.488 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.484+0000 7fb4cdee8700 1 -- 172.21.15.58:0/300707387 shutdown_connections 2021-11-20T05:33:20.489 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.484+0000 7fb4cdee8700 1 --2- 172.21.15.58:0/300707387 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb4b405c2a0 0x7fb4b405e760 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:20.489 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.484+0000 7fb4cdee8700 1 --2- 172.21.15.58:0/300707387 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb4c80f4aa0 0x7fb4c8107100 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:20.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.484+0000 7fb4cdee8700 1 --2- 172.21.15.58:0/300707387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4c80f5440 0x7fb4c81075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:20.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.484+0000 7fb4cdee8700 1 -- 172.21.15.58:0/300707387 >> 172.21.15.58:0/300707387 conn(0x7fb4c80f05e0 msgr2=0x7fb4c80fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:20.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.484+0000 7fb4cdee8700 1 -- 172.21.15.58:0/300707387 shutdown_connections 2021-11-20T05:33:20.491 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:20.484+0000 7fb4cdee8700 1 -- 172.21.15.58:0/300707387 wait complete. 2021-11-20T05:33:20.880 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2021-11-20T05:33:21.323 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:21.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:21 smithi058 conmon[33537]: cluster 2021-11-20T05:33:20 2021-11-20T05:33:21.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:21 smithi058 conmon[33537]: .200991+0000 mgr.smithi058.zagdlk (mgr.14164) 156 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:21.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:21 smithi058 conmon[33537]: audit 2021-11-20T05:33:20.483121+0000 mon.smithi058 (mon.0) 435 : audit [DBG] from='client.? 172.21.15.58:0/300707387' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T05:33:21.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:21 smithi187 conmon[36358]: cluster 2021-11-20T05:33:20.200991+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:33:21.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:21 smithi187 conmon[36358]: mgr.14164) 156 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:21.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:21 smithi187 conmon[36358]: audit 2021-11-20T05:33:20.483121+0000 mon.smithi058 (mon.0) 435 : audit [DBG] from='client.? 172.21.15.58:0/300707387' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T05:33:22.495 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.493+0000 7fab24d88700 1 -- 172.21.15.58:0/1760283971 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab200f5520 msgr2=0x7fab200f5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:22.496 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.493+0000 7fab24d88700 1 --2- 172.21.15.58:0/1760283971 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab200f5520 0x7fab200f5940 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fab14004660 tx=0x7fab14009b30).stop 2021-11-20T05:33:22.496 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.493+0000 7fab24d88700 1 -- 172.21.15.58:0/1760283971 shutdown_connections 2021-11-20T05:33:22.496 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.493+0000 7fab24d88700 1 --2- 172.21.15.58:0/1760283971 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab200f67c0 0x7fab200f6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:22.496 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.493+0000 7fab24d88700 1 --2- 172.21.15.58:0/1760283971 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab200f5520 0x7fab200f5940 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:22.496 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.493+0000 7fab24d88700 1 -- 172.21.15.58:0/1760283971 >> 172.21.15.58:0/1760283971 conn(0x7fab200f0ca0 msgr2=0x7fab200f30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:22.497 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.494+0000 7fab24d88700 1 -- 172.21.15.58:0/1760283971 shutdown_connections 2021-11-20T05:33:22.497 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.494+0000 7fab24d88700 1 -- 172.21.15.58:0/1760283971 wait complete. 2021-11-20T05:33:22.498 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.494+0000 7fab24d88700 1 Processor -- start 2021-11-20T05:33:22.498 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.494+0000 7fab24d88700 1 -- start start 2021-11-20T05:33:22.498 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.494+0000 7fab24d88700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab200f5520 0x7fab20106f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:22.499 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.494+0000 7fab24d88700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab200f67c0 0x7fab20107400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:22.499 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.494+0000 7fab24d88700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab201079b0 con 0x7fab200f5520 2021-11-20T05:33:22.499 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.494+0000 7fab24d88700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab20107af0 con 0x7fab200f67c0 2021-11-20T05:33:22.499 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.495+0000 7fab1f7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab200f5520 0x7fab20106f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:22.500 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.495+0000 7fab1f7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab200f5520 0x7fab20106f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40266/0 (socket says 172.21.15.58:40266) 2021-11-20T05:33:22.500 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.495+0000 7fab1f7fe700 1 -- 172.21.15.58:0/3330740450 learned_addr learned my addr 172.21.15.58:0/3330740450 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:22.500 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.495+0000 7fab1effd700 1 --2- 172.21.15.58:0/3330740450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab200f67c0 0x7fab20107400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:22.500 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.495+0000 7fab1f7fe700 1 -- 172.21.15.58:0/3330740450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab200f67c0 msgr2=0x7fab20107400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:22.501 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.495+0000 7fab1f7fe700 1 --2- 172.21.15.58:0/3330740450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab200f67c0 0x7fab20107400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:22.501 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.495+0000 7fab1f7fe700 1 -- 172.21.15.58:0/3330740450 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab14005040 con 0x7fab200f5520 2021-11-20T05:33:22.501 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.495+0000 7fab1f7fe700 1 --2- 172.21.15.58:0/3330740450 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab200f5520 0x7fab20106f10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fab14004e30 tx=0x7fab14009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:22.501 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.495+0000 7fab07fff700 1 -- 172.21.15.58:0/3330740450 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab14018070 con 0x7fab200f5520 2021-11-20T05:33:22.502 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.495+0000 7fab07fff700 1 -- 172.21.15.58:0/3330740450 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab14013410 con 0x7fab200f5520 2021-11-20T05:33:22.502 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.495+0000 7fab07fff700 1 -- 172.21.15.58:0/3330740450 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab1401c3c0 con 0x7fab200f5520 2021-11-20T05:33:22.502 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.495+0000 7fab24d88700 1 -- 172.21.15.58:0/3330740450 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab20108430 con 0x7fab200f5520 2021-11-20T05:33:22.502 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.495+0000 7fab24d88700 1 -- 172.21.15.58:0/3330740450 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab20108940 con 0x7fab200f5520 2021-11-20T05:33:22.503 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.496+0000 7fab24d88700 1 -- 172.21.15.58:0/3330740450 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fab2005db60 con 0x7fab200f5520 2021-11-20T05:33:22.504 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.497+0000 7fab07fff700 1 -- 172.21.15.58:0/3330740450 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fab1401d420 con 0x7fab200f5520 2021-11-20T05:33:22.504 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.497+0000 7fab07fff700 1 --2- 172.21.15.58:0/3330740450 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fab0805c2b0 0x7fab0805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:22.504 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.497+0000 7fab07fff700 1 -- 172.21.15.58:0/3330740450 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fab14077780 con 0x7fab200f5520 2021-11-20T05:33:22.505 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.497+0000 7fab1effd700 1 --2- 172.21.15.58:0/3330740450 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fab0805c2b0 0x7fab0805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:22.505 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.497+0000 7fab07fff700 1 --2- 172.21.15.58:0/3330740450 >> [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844] conn(0x7fab08062040 0x7fab080644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:22.505 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.497+0000 7fab07fff700 1 -- 172.21.15.58:0/3330740450 --> [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab08064b20 con 0x7fab08062040 2021-11-20T05:33:22.505 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.497+0000 7fab07fff700 1 -- 172.21.15.58:0/3330740450 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab1407c220 con 0x7fab200f5520 2021-11-20T05:33:22.506 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.497+0000 7fab1ffff700 1 --2- 172.21.15.58:0/3330740450 >> [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844] conn(0x7fab08062040 0x7fab080644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:22.506 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.497+0000 7fab1effd700 1 --2- 172.21.15.58:0/3330740450 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fab0805c2b0 0x7fab0805e770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fab100096c0 tx=0x7fab10009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:22.506 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.498+0000 7fab1ffff700 1 --2- 172.21.15.58:0/3330740450 >> [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844] conn(0x7fab08062040 0x7fab080644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:22.506 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.498+0000 7fab07fff700 1 -- 172.21.15.58:0/3330740450 <== osd.0 v2:172.21.15.58:6802/2375873844 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fab08064b20 con 0x7fab08062040 2021-11-20T05:33:22.512 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.513+0000 7fab24d88700 1 -- 172.21.15.58:0/3330740450 --> [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fab20108df0 con 0x7fab08062040 2021-11-20T05:33:22.512 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.513+0000 7fab07fff700 1 -- 172.21.15.58:0/3330740450 <== osd.0 v2:172.21.15.58:6802/2375873844 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fab20108df0 con 0x7fab08062040 2021-11-20T05:33:22.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.513+0000 7fab24d88700 1 -- 172.21.15.58:0/3330740450 >> [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844] conn(0x7fab08062040 msgr2=0x7fab080644a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:22.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.513+0000 7fab24d88700 1 --2- 172.21.15.58:0/3330740450 >> [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844] conn(0x7fab08062040 0x7fab080644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:22.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.513+0000 7fab24d88700 1 -- 172.21.15.58:0/3330740450 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fab0805c2b0 msgr2=0x7fab0805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:22.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.513+0000 7fab24d88700 1 --2- 172.21.15.58:0/3330740450 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fab0805c2b0 0x7fab0805e770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fab100096c0 tx=0x7fab10009be0).stop 2021-11-20T05:33:22.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.513+0000 7fab24d88700 1 -- 172.21.15.58:0/3330740450 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab200f5520 msgr2=0x7fab20106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:22.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.513+0000 7fab24d88700 1 --2- 172.21.15.58:0/3330740450 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab200f5520 0x7fab20106f10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fab14004e30 tx=0x7fab14009b30).stop 2021-11-20T05:33:22.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.513+0000 7fab24d88700 1 -- 172.21.15.58:0/3330740450 shutdown_connections 2021-11-20T05:33:22.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.513+0000 7fab24d88700 1 --2- 172.21.15.58:0/3330740450 >> [v2:172.21.15.58:6802/2375873844,v1:172.21.15.58:6803/2375873844] conn(0x7fab08062040 0x7fab080644a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:22.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.513+0000 7fab24d88700 1 --2- 172.21.15.58:0/3330740450 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fab0805c2b0 0x7fab0805e770 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:22.516 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.513+0000 7fab24d88700 1 --2- 172.21.15.58:0/3330740450 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fab200f5520 0x7fab20106f10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:22.516 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.513+0000 7fab24d88700 1 --2- 172.21.15.58:0/3330740450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab200f67c0 0x7fab20107400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:22.516 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.513+0000 7fab24d88700 1 -- 172.21.15.58:0/3330740450 >> 172.21.15.58:0/3330740450 conn(0x7fab200f0ca0 msgr2=0x7fab2005fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:22.516 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.514+0000 7fab24d88700 1 -- 172.21.15.58:0/3330740450 shutdown_connections 2021-11-20T05:33:22.517 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:22.514+0000 7fab24d88700 1 -- 172.21.15.58:0/3330740450 wait complete. 2021-11-20T05:33:22.839 INFO:teuthology.orchestra.run.smithi058.stdout:38654705687 2021-11-20T05:33:22.839 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2021-11-20T05:33:23.284 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:23.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:23 smithi058 conmon[33537]: cluster 2021-11-20T05:33:22.201288+0000 mgr.smithi058.zagdlk 2021-11-20T05:33:23.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:23 smithi058 conmon[33537]: (mgr.14164) 157 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:23.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:23 smithi187 conmon[36358]: cluster 2021-11-20T05:33:22.201288+0000 mgr.smithi058.zagdlk (mgr.14164) 157 2021-11-20T05:33:23.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:23 smithi187 conmon[36358]: : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:24.432 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.430+0000 7fa4c1b85700 1 -- 172.21.15.58:0/1709064828 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa4bc0f7ce0 msgr2=0x7fa4bc0f8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:24.432 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.430+0000 7fa4c1b85700 1 --2- 172.21.15.58:0/1709064828 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa4bc0f7ce0 0x7fa4bc0f8100 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fa4b0004660 tx=0x7fa4b0009b30).stop 2021-11-20T05:33:24.433 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.430+0000 7fa4c1b85700 1 -- 172.21.15.58:0/1709064828 shutdown_connections 2021-11-20T05:33:24.433 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.430+0000 7fa4c1b85700 1 --2- 172.21.15.58:0/1709064828 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4bc0f62f0 0x7fa4bc0f6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:24.433 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.430+0000 7fa4c1b85700 1 --2- 172.21.15.58:0/1709064828 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa4bc0f7ce0 0x7fa4bc0f8100 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:24.433 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.430+0000 7fa4c1b85700 1 -- 172.21.15.58:0/1709064828 >> 172.21.15.58:0/1709064828 conn(0x7fa4bc0f1d70 msgr2=0x7fa4bc0f41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:24.434 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.430+0000 7fa4c1b85700 1 -- 172.21.15.58:0/1709064828 shutdown_connections 2021-11-20T05:33:24.434 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.430+0000 7fa4c1b85700 1 -- 172.21.15.58:0/1709064828 wait complete. 2021-11-20T05:33:24.434 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.431+0000 7fa4c1b85700 1 Processor -- start 2021-11-20T05:33:24.434 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.431+0000 7fa4c1b85700 1 -- start start 2021-11-20T05:33:24.435 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.431+0000 7fa4c1b85700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4bc0f62f0 0x7fa4bc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:24.435 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.431+0000 7fa4c1b85700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa4bc0f7ce0 0x7fa4bc069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:24.435 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.431+0000 7fa4c1b85700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc06a380 con 0x7fa4bc0f7ce0 2021-11-20T05:33:24.435 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.431+0000 7fa4c1b85700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc06a4c0 con 0x7fa4bc0f62f0 2021-11-20T05:33:24.436 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.431+0000 7fa4bbfff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa4bc0f7ce0 0x7fa4bc069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:24.436 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.431+0000 7fa4bbfff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa4bc0f7ce0 0x7fa4bc069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40278/0 (socket says 172.21.15.58:40278) 2021-11-20T05:33:24.436 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.432+0000 7fa4bbfff700 1 -- 172.21.15.58:0/3653901558 learned_addr learned my addr 172.21.15.58:0/3653901558 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:24.436 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.432+0000 7fa4c0b83700 1 --2- 172.21.15.58:0/3653901558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4bc0f62f0 0x7fa4bc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:24.437 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.432+0000 7fa4bbfff700 1 -- 172.21.15.58:0/3653901558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4bc0f62f0 msgr2=0x7fa4bc06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:24.437 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.432+0000 7fa4bbfff700 1 --2- 172.21.15.58:0/3653901558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4bc0f62f0 0x7fa4bc06b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:24.437 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.432+0000 7fa4bbfff700 1 -- 172.21.15.58:0/3653901558 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b0005040 con 0x7fa4bc0f7ce0 2021-11-20T05:33:24.437 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.432+0000 7fa4bbfff700 1 --2- 172.21.15.58:0/3653901558 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa4bc0f7ce0 0x7fa4bc069e20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa4a80026d0 tx=0x7fa4a8009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:24.438 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.432+0000 7fa4b97fa700 1 -- 172.21.15.58:0/3653901558 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4a80079c0 con 0x7fa4bc0f7ce0 2021-11-20T05:33:24.438 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.432+0000 7fa4b97fa700 1 -- 172.21.15.58:0/3653901558 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4a800f040 con 0x7fa4bc0f7ce0 2021-11-20T05:33:24.438 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.432+0000 7fa4c1b85700 1 -- 172.21.15.58:0/3653901558 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4bc06a7a0 con 0x7fa4bc0f7ce0 2021-11-20T05:33:24.438 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.432+0000 7fa4b97fa700 1 -- 172.21.15.58:0/3653901558 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4a80148b0 con 0x7fa4bc0f7ce0 2021-11-20T05:33:24.439 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.432+0000 7fa4c1b85700 1 -- 172.21.15.58:0/3653901558 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4bc10f000 con 0x7fa4bc0f7ce0 2021-11-20T05:33:24.440 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.433+0000 7fa4c1b85700 1 -- 172.21.15.58:0/3653901558 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa4bc059100 con 0x7fa4bc0f7ce0 2021-11-20T05:33:24.441 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.433+0000 7fa4b97fa700 1 -- 172.21.15.58:0/3653901558 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa4a8007b20 con 0x7fa4bc0f7ce0 2021-11-20T05:33:24.441 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.434+0000 7fa4b97fa700 1 --2- 172.21.15.58:0/3653901558 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa4ac05c2b0 0x7fa4ac05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:24.441 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.434+0000 7fa4b97fa700 1 -- 172.21.15.58:0/3653901558 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fa4a8075a80 con 0x7fa4bc0f7ce0 2021-11-20T05:33:24.441 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.434+0000 7fa4c0b83700 1 --2- 172.21.15.58:0/3653901558 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa4ac05c2b0 0x7fa4ac05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:24.441 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.434+0000 7fa4b97fa700 1 --2- 172.21.15.58:0/3653901558 >> [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869] conn(0x7fa4ac062040 0x7fa4ac0644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:24.442 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.434+0000 7fa4b97fa700 1 -- 172.21.15.58:0/3653901558 --> [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4ac064b20 con 0x7fa4ac062040 2021-11-20T05:33:24.442 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.434+0000 7fa4b97fa700 1 -- 172.21.15.58:0/3653901558 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4a807a220 con 0x7fa4bc0f7ce0 2021-11-20T05:33:24.442 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.434+0000 7fa4c1384700 1 --2- 172.21.15.58:0/3653901558 >> [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869] conn(0x7fa4ac062040 0x7fa4ac0644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:24.442 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.434+0000 7fa4c0b83700 1 --2- 172.21.15.58:0/3653901558 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa4ac05c2b0 0x7fa4ac05e770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fa4bc06b730 tx=0x7fa4b0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:24.443 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.434+0000 7fa4c1384700 1 --2- 172.21.15.58:0/3653901558 >> [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869] conn(0x7fa4ac062040 0x7fa4ac0644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:24.443 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.435+0000 7fa4b97fa700 1 -- 172.21.15.58:0/3653901558 <== osd.1 v2:172.21.15.58:6810/3875723869 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fa4ac064b20 con 0x7fa4ac062040 2021-11-20T05:33:24.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.449+0000 7fa4c1b85700 1 -- 172.21.15.58:0/3653901558 --> [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa4bc0581d0 con 0x7fa4ac062040 2021-11-20T05:33:24.450 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.450+0000 7fa4b97fa700 1 -- 172.21.15.58:0/3653901558 <== osd.1 v2:172.21.15.58:6810/3875723869 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa4bc0581d0 con 0x7fa4ac062040 2021-11-20T05:33:24.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.450+0000 7fa4c1b85700 1 -- 172.21.15.58:0/3653901558 >> [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869] conn(0x7fa4ac062040 msgr2=0x7fa4ac0644a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:24.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.450+0000 7fa4c1b85700 1 --2- 172.21.15.58:0/3653901558 >> [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869] conn(0x7fa4ac062040 0x7fa4ac0644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:24.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.450+0000 7fa4c1b85700 1 -- 172.21.15.58:0/3653901558 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa4ac05c2b0 msgr2=0x7fa4ac05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:24.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.450+0000 7fa4c1b85700 1 --2- 172.21.15.58:0/3653901558 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa4ac05c2b0 0x7fa4ac05e770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fa4bc06b730 tx=0x7fa4b0009b30).stop 2021-11-20T05:33:24.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.450+0000 7fa4c1b85700 1 -- 172.21.15.58:0/3653901558 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa4bc0f7ce0 msgr2=0x7fa4bc069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:24.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.450+0000 7fa4c1b85700 1 --2- 172.21.15.58:0/3653901558 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa4bc0f7ce0 0x7fa4bc069e20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa4a80026d0 tx=0x7fa4a8009de0).stop 2021-11-20T05:33:24.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.450+0000 7fa4c1b85700 1 -- 172.21.15.58:0/3653901558 shutdown_connections 2021-11-20T05:33:24.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.450+0000 7fa4c1b85700 1 --2- 172.21.15.58:0/3653901558 >> [v2:172.21.15.58:6810/3875723869,v1:172.21.15.58:6811/3875723869] conn(0x7fa4ac062040 0x7fa4ac0644a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:24.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.450+0000 7fa4c1b85700 1 --2- 172.21.15.58:0/3653901558 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa4ac05c2b0 0x7fa4ac05e770 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:24.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.450+0000 7fa4c1b85700 1 --2- 172.21.15.58:0/3653901558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa4bc0f62f0 0x7fa4bc06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:24.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.450+0000 7fa4c1b85700 1 --2- 172.21.15.58:0/3653901558 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa4bc0f7ce0 0x7fa4bc069e20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:24.454 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.450+0000 7fa4c1b85700 1 -- 172.21.15.58:0/3653901558 >> 172.21.15.58:0/3653901558 conn(0x7fa4bc0f1d70 msgr2=0x7fa4bc0fb320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:24.454 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.450+0000 7fa4c1b85700 1 -- 172.21.15.58:0/3653901558 shutdown_connections 2021-11-20T05:33:24.454 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:24.450+0000 7fa4c1b85700 1 -- 172.21.15.58:0/3653901558 wait complete. 2021-11-20T05:33:24.795 INFO:teuthology.orchestra.run.smithi058.stdout:60129542165 2021-11-20T05:33:24.796 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2021-11-20T05:33:25.240 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:25.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:25 smithi058 conmon[33537]: cluster 2021-11-20T05:33:24.201588+0000 mgr.smithi058.zagdlk (mgr.14164) 158 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:25.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:25 smithi187 conmon[36358]: cluster 2021-11-20T05:33:24.201588+0000 mgr.smithi058.zagdlk (mgr.14164) 158 : 2021-11-20T05:33:25.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:25 smithi187 conmon[36358]: cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:26.432 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.430+0000 7efd20181700 1 -- 172.21.15.58:0/4151211947 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efd180f6990 msgr2=0x7efd180f6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:26.432 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.430+0000 7efd20181700 1 --2- 172.21.15.58:0/4151211947 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efd180f6990 0x7efd180f6df0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7efd08004660 tx=0x7efd08009b30).stop 2021-11-20T05:33:26.432 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.430+0000 7efd20181700 1 -- 172.21.15.58:0/4151211947 shutdown_connections 2021-11-20T05:33:26.432 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.430+0000 7efd20181700 1 --2- 172.21.15.58:0/4151211947 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efd180f6990 0x7efd180f6df0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:26.433 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.430+0000 7efd20181700 1 --2- 172.21.15.58:0/4151211947 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd180f56f0 0x7efd180f5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:26.433 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.430+0000 7efd20181700 1 -- 172.21.15.58:0/4151211947 >> 172.21.15.58:0/4151211947 conn(0x7efd180f0e50 msgr2=0x7efd180f3270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:26.433 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.430+0000 7efd20181700 1 -- 172.21.15.58:0/4151211947 shutdown_connections 2021-11-20T05:33:26.434 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.430+0000 7efd20181700 1 -- 172.21.15.58:0/4151211947 wait complete. 2021-11-20T05:33:26.434 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.431+0000 7efd20181700 1 Processor -- start 2021-11-20T05:33:26.434 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.431+0000 7efd20181700 1 -- start start 2021-11-20T05:33:26.435 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.431+0000 7efd20181700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd180f56f0 0x7efd18109db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:26.435 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.431+0000 7efd20181700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efd180f6990 0x7efd1810c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:26.435 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.431+0000 7efd20181700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd18060940 con 0x7efd180f6990 2021-11-20T05:33:26.436 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.431+0000 7efd20181700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd18060a80 con 0x7efd180f56f0 2021-11-20T05:33:26.436 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.431+0000 7efd1d71c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efd180f6990 0x7efd1810c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:26.436 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.431+0000 7efd1d71c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efd180f6990 0x7efd1810c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40290/0 (socket says 172.21.15.58:40290) 2021-11-20T05:33:26.436 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.431+0000 7efd1d71c700 1 -- 172.21.15.58:0/4131916751 learned_addr learned my addr 172.21.15.58:0/4131916751 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:26.437 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.431+0000 7efd1df1d700 1 --2- 172.21.15.58:0/4131916751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd180f56f0 0x7efd18109db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:26.437 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.431+0000 7efd1d71c700 1 -- 172.21.15.58:0/4131916751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd180f56f0 msgr2=0x7efd18109db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:26.437 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.432+0000 7efd1d71c700 1 --2- 172.21.15.58:0/4131916751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd180f56f0 0x7efd18109db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:26.438 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.432+0000 7efd1d71c700 1 -- 172.21.15.58:0/4131916751 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd08005040 con 0x7efd180f6990 2021-11-20T05:33:26.438 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.432+0000 7efd1df1d700 1 --2- 172.21.15.58:0/4131916751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd180f56f0 0x7efd18109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:33:26.438 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.432+0000 7efd1d71c700 1 --2- 172.21.15.58:0/4131916751 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efd180f6990 0x7efd1810c2b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7efd08009830 tx=0x7efd08009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:26.439 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.432+0000 7efd0e7fc700 1 -- 172.21.15.58:0/4131916751 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd08018070 con 0x7efd180f6990 2021-11-20T05:33:26.439 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.432+0000 7efd20181700 1 -- 172.21.15.58:0/4131916751 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd1810c800 con 0x7efd180f6990 2021-11-20T05:33:26.439 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.432+0000 7efd0e7fc700 1 -- 172.21.15.58:0/4131916751 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd08013410 con 0x7efd180f6990 2021-11-20T05:33:26.440 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.432+0000 7efd20181700 1 -- 172.21.15.58:0/4131916751 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd1810cd60 con 0x7efd180f6990 2021-11-20T05:33:26.440 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.432+0000 7efd0e7fc700 1 -- 172.21.15.58:0/4131916751 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd0801c620 con 0x7efd180f6990 2021-11-20T05:33:26.442 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.433+0000 7efd20181700 1 -- 172.21.15.58:0/4131916751 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efd1805dcf0 con 0x7efd180f6990 2021-11-20T05:33:26.442 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.433+0000 7efd0e7fc700 1 -- 172.21.15.58:0/4131916751 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7efd0801c8e0 con 0x7efd180f6990 2021-11-20T05:33:26.443 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.433+0000 7efd0e7fc700 1 --2- 172.21.15.58:0/4131916751 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efd0405c2a0 0x7efd0405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:26.443 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.434+0000 7efd0e7fc700 1 -- 172.21.15.58:0/4131916751 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7efd08077a70 con 0x7efd180f6990 2021-11-20T05:33:26.443 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.434+0000 7efd1df1d700 1 --2- 172.21.15.58:0/4131916751 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efd0405c2a0 0x7efd0405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:26.443 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.434+0000 7efd0e7fc700 1 --2- 172.21.15.58:0/4131916751 >> [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076] conn(0x7efd04062080 0x7efd040644e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:26.444 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.434+0000 7efd0e7fc700 1 -- 172.21.15.58:0/4131916751 --> [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd04064b60 con 0x7efd04062080 2021-11-20T05:33:26.444 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.434+0000 7efd0e7fc700 1 -- 172.21.15.58:0/4131916751 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd0807c220 con 0x7efd180f6990 2021-11-20T05:33:26.444 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.434+0000 7efd1e71e700 1 --2- 172.21.15.58:0/4131916751 >> [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076] conn(0x7efd04062080 0x7efd040644e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:26.444 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.434+0000 7efd1df1d700 1 --2- 172.21.15.58:0/4131916751 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efd0405c2a0 0x7efd0405e760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7efd180605c0 tx=0x7efd14009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:26.445 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.434+0000 7efd1e71e700 1 --2- 172.21.15.58:0/4131916751 >> [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076] conn(0x7efd04062080 0x7efd040644e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:26.445 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.435+0000 7efd0e7fc700 1 -- 172.21.15.58:0/4131916751 <== osd.2 v2:172.21.15.58:6818/745015076 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7efd04064b60 con 0x7efd04062080 2021-11-20T05:33:26.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.449+0000 7efd20181700 1 -- 172.21.15.58:0/4131916751 --> [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efd180581d0 con 0x7efd04062080 2021-11-20T05:33:26.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.449+0000 7efd0e7fc700 1 -- 172.21.15.58:0/4131916751 <== osd.2 v2:172.21.15.58:6818/745015076 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efd180581d0 con 0x7efd04062080 2021-11-20T05:33:26.450 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.450+0000 7efd20181700 1 -- 172.21.15.58:0/4131916751 >> [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076] conn(0x7efd04062080 msgr2=0x7efd040644e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:26.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.450+0000 7efd20181700 1 --2- 172.21.15.58:0/4131916751 >> [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076] conn(0x7efd04062080 0x7efd040644e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:26.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.450+0000 7efd20181700 1 -- 172.21.15.58:0/4131916751 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efd0405c2a0 msgr2=0x7efd0405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:26.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.450+0000 7efd20181700 1 --2- 172.21.15.58:0/4131916751 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efd0405c2a0 0x7efd0405e760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7efd180605c0 tx=0x7efd14009040).stop 2021-11-20T05:33:26.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.450+0000 7efd20181700 1 -- 172.21.15.58:0/4131916751 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efd180f6990 msgr2=0x7efd1810c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:26.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.450+0000 7efd20181700 1 --2- 172.21.15.58:0/4131916751 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efd180f6990 0x7efd1810c2b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7efd08009830 tx=0x7efd08009b30).stop 2021-11-20T05:33:26.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.450+0000 7efd20181700 1 -- 172.21.15.58:0/4131916751 shutdown_connections 2021-11-20T05:33:26.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.450+0000 7efd20181700 1 --2- 172.21.15.58:0/4131916751 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efd0405c2a0 0x7efd0405e760 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:26.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.450+0000 7efd20181700 1 --2- 172.21.15.58:0/4131916751 >> [v2:172.21.15.58:6818/745015076,v1:172.21.15.58:6819/745015076] conn(0x7efd04062080 0x7efd040644e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:26.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.450+0000 7efd20181700 1 --2- 172.21.15.58:0/4131916751 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd180f56f0 0x7efd18109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:26.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.450+0000 7efd20181700 1 --2- 172.21.15.58:0/4131916751 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efd180f6990 0x7efd1810c2b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:26.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.450+0000 7efd20181700 1 -- 172.21.15.58:0/4131916751 >> 172.21.15.58:0/4131916751 conn(0x7efd180f0e50 msgr2=0x7efd180fa3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:26.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.450+0000 7efd20181700 1 -- 172.21.15.58:0/4131916751 shutdown_connections 2021-11-20T05:33:26.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:26.450+0000 7efd20181700 1 -- 172.21.15.58:0/4131916751 wait complete. 2021-11-20T05:33:26.953 INFO:teuthology.orchestra.run.smithi058.stdout:81604378639 2021-11-20T05:33:26.954 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2021-11-20T05:33:27.400 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:27.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:27 smithi058 conmon[33537]: cluster 2021-11-20T05:33:26.202005+0000 mgr.smithi058.zagdlk (mgr.14164) 159 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:27.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:27 smithi187 conmon[36358]: cluster 2021-11-20T05:33:26.202005+0000 mgr.smithi058.zagdlk (mgr.14164) 159 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:28.547 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.545+0000 7f3692955700 1 -- 172.21.15.58:0/1964261715 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f368c0f9800 msgr2=0x7f368c0fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:28.547 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.545+0000 7f3692955700 1 --2- 172.21.15.58:0/1964261715 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f368c0f9800 0x7f368c0fbbf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f367c004660 tx=0x7f367c009b30).stop 2021-11-20T05:33:28.547 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.545+0000 7f3692955700 1 -- 172.21.15.58:0/1964261715 shutdown_connections 2021-11-20T05:33:28.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.545+0000 7f3692955700 1 --2- 172.21.15.58:0/1964261715 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f368c0f9800 0x7f368c0fbbf0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:28.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.545+0000 7f3692955700 1 --2- 172.21.15.58:0/1964261715 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f368c0607f0 0x7f368c0f9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:28.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.545+0000 7f3692955700 1 -- 172.21.15.58:0/1964261715 >> 172.21.15.58:0/1964261715 conn(0x7f368c0f0ca0 msgr2=0x7f368c0f30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:28.549 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.546+0000 7f3692955700 1 -- 172.21.15.58:0/1964261715 shutdown_connections 2021-11-20T05:33:28.549 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.546+0000 7f3692955700 1 -- 172.21.15.58:0/1964261715 wait complete. 2021-11-20T05:33:28.549 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.546+0000 7f3692955700 1 Processor -- start 2021-11-20T05:33:28.550 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.546+0000 7f3692955700 1 -- start start 2021-11-20T05:33:28.550 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.546+0000 7f3692955700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f368c0607f0 0x7f368c0f6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:28.550 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.546+0000 7f3692955700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f368c0f9800 0x7f368c0f5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:28.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.546+0000 7f3692955700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368c0f56e0 con 0x7f368c0607f0 2021-11-20T05:33:28.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.546+0000 7f3692955700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368c0f5820 con 0x7f368c0f9800 2021-11-20T05:33:28.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.546+0000 7f3691953700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f368c0607f0 0x7f368c0f6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:28.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.546+0000 7f3691953700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f368c0607f0 0x7f368c0f6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40300/0 (socket says 172.21.15.58:40300) 2021-11-20T05:33:28.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.546+0000 7f3691953700 1 -- 172.21.15.58:0/1864786111 learned_addr learned my addr 172.21.15.58:0/1864786111 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:28.552 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.547+0000 7f3691152700 1 --2- 172.21.15.58:0/1864786111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f368c0f9800 0x7f368c0f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:28.552 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.547+0000 7f3691953700 1 -- 172.21.15.58:0/1864786111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f368c0f9800 msgr2=0x7f368c0f5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:28.552 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.547+0000 7f3691953700 1 --2- 172.21.15.58:0/1864786111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f368c0f9800 0x7f368c0f5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:28.553 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.547+0000 7f3691953700 1 -- 172.21.15.58:0/1864786111 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367c005040 con 0x7f368c0607f0 2021-11-20T05:33:28.553 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.547+0000 7f3691953700 1 --2- 172.21.15.58:0/1864786111 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f368c0607f0 0x7f368c0f6d00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f36880026d0 tx=0x7f3688009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:28.553 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.547+0000 7f36827fc700 1 -- 172.21.15.58:0/1864786111 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36880079c0 con 0x7f368c0607f0 2021-11-20T05:33:28.553 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.547+0000 7f3692955700 1 -- 172.21.15.58:0/1864786111 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f368c0f5b00 con 0x7f368c0607f0 2021-11-20T05:33:28.554 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.547+0000 7f36827fc700 1 -- 172.21.15.58:0/1864786111 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f368800f040 con 0x7f368c0607f0 2021-11-20T05:33:28.554 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.547+0000 7f36827fc700 1 -- 172.21.15.58:0/1864786111 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36880148b0 con 0x7f368c0607f0 2021-11-20T05:33:28.554 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.547+0000 7f3692955700 1 -- 172.21.15.58:0/1864786111 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f368c10ef50 con 0x7f368c0607f0 2021-11-20T05:33:28.556 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.548+0000 7f3692955700 1 -- 172.21.15.58:0/1864786111 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f368c05db60 con 0x7f368c0607f0 2021-11-20T05:33:28.556 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.548+0000 7f36827fc700 1 -- 172.21.15.58:0/1864786111 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3688007b20 con 0x7f368c0607f0 2021-11-20T05:33:28.556 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.549+0000 7f36827fc700 1 --2- 172.21.15.58:0/1864786111 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f367805c2b0 0x7f367805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:28.556 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.549+0000 7f36827fc700 1 -- 172.21.15.58:0/1864786111 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f3688075b00 con 0x7f368c0607f0 2021-11-20T05:33:28.557 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.549+0000 7f3691152700 1 --2- 172.21.15.58:0/1864786111 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f367805c2b0 0x7f367805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:28.557 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.549+0000 7f36827fc700 1 --2- 172.21.15.58:0/1864786111 >> [v2:172.21.15.187:6800/1941886381,v1:172.21.15.187:6801/1941886381] conn(0x7f3678062040 0x7f36780644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:28.557 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.549+0000 7f36827fc700 1 -- 172.21.15.58:0/1864786111 --> [v2:172.21.15.187:6800/1941886381,v1:172.21.15.187:6801/1941886381] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3678064b20 con 0x7f3678062040 2021-11-20T05:33:28.557 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.549+0000 7f36827fc700 1 -- 172.21.15.58:0/1864786111 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f368807a220 con 0x7f368c0607f0 2021-11-20T05:33:28.558 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.549+0000 7f3692154700 1 --2- 172.21.15.58:0/1864786111 >> [v2:172.21.15.187:6800/1941886381,v1:172.21.15.187:6801/1941886381] conn(0x7f3678062040 0x7f36780644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:28.558 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.549+0000 7f3691152700 1 --2- 172.21.15.58:0/1864786111 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f367805c2b0 0x7f367805e770 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f367c009f50 tx=0x7f367c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:28.558 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.550+0000 7f3692154700 1 --2- 172.21.15.58:0/1864786111 >> [v2:172.21.15.187:6800/1941886381,v1:172.21.15.187:6801/1941886381] conn(0x7f3678062040 0x7f36780644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:28.558 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.550+0000 7f36827fc700 1 -- 172.21.15.58:0/1864786111 <== osd.3 v2:172.21.15.187:6800/1941886381 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f3678064b20 con 0x7f3678062040 2021-11-20T05:33:28.564 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.565+0000 7f3692955700 1 -- 172.21.15.58:0/1864786111 --> [v2:172.21.15.187:6800/1941886381,v1:172.21.15.187:6801/1941886381] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f368c058150 con 0x7f3678062040 2021-11-20T05:33:28.565 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.565+0000 7f36827fc700 1 -- 172.21.15.58:0/1864786111 <== osd.3 v2:172.21.15.187:6800/1941886381 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f368c058150 con 0x7f3678062040 2021-11-20T05:33:28.566 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.566+0000 7f3692955700 1 -- 172.21.15.58:0/1864786111 >> [v2:172.21.15.187:6800/1941886381,v1:172.21.15.187:6801/1941886381] conn(0x7f3678062040 msgr2=0x7f36780644a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:28.566 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.566+0000 7f3692955700 1 --2- 172.21.15.58:0/1864786111 >> [v2:172.21.15.187:6800/1941886381,v1:172.21.15.187:6801/1941886381] conn(0x7f3678062040 0x7f36780644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:28.566 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.566+0000 7f3692955700 1 -- 172.21.15.58:0/1864786111 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f367805c2b0 msgr2=0x7f367805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:28.567 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.566+0000 7f3692955700 1 --2- 172.21.15.58:0/1864786111 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f367805c2b0 0x7f367805e770 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f367c009f50 tx=0x7f367c009b30).stop 2021-11-20T05:33:28.567 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.566+0000 7f3692955700 1 -- 172.21.15.58:0/1864786111 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f368c0607f0 msgr2=0x7f368c0f6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:28.567 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.566+0000 7f3692955700 1 --2- 172.21.15.58:0/1864786111 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f368c0607f0 0x7f368c0f6d00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f36880026d0 tx=0x7f3688009de0).stop 2021-11-20T05:33:28.567 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.566+0000 7f3692955700 1 -- 172.21.15.58:0/1864786111 shutdown_connections 2021-11-20T05:33:28.568 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.566+0000 7f3692955700 1 --2- 172.21.15.58:0/1864786111 >> [v2:172.21.15.187:6800/1941886381,v1:172.21.15.187:6801/1941886381] conn(0x7f3678062040 0x7f36780644a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:28.568 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.566+0000 7f3692955700 1 --2- 172.21.15.58:0/1864786111 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f367805c2b0 0x7f367805e770 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:28.568 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.566+0000 7f3692955700 1 --2- 172.21.15.58:0/1864786111 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f368c0607f0 0x7f368c0f6d00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:28.568 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.566+0000 7f3692955700 1 --2- 172.21.15.58:0/1864786111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f368c0f9800 0x7f368c0f5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:28.568 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.566+0000 7f3692955700 1 -- 172.21.15.58:0/1864786111 >> 172.21.15.58:0/1864786111 conn(0x7f368c0f0ca0 msgr2=0x7f368c0f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:28.569 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.566+0000 7f3692955700 1 -- 172.21.15.58:0/1864786111 shutdown_connections 2021-11-20T05:33:28.569 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:28.566+0000 7f3692955700 1 -- 172.21.15.58:0/1864786111 wait complete. 2021-11-20T05:33:28.903 INFO:teuthology.orchestra.run.smithi058.stdout:111669149709 2021-11-20T05:33:28.904 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2021-11-20T05:33:29.348 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:29.515 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:29 smithi058 conmon[33537]: cluster 2021-11-20T05:33:28.202286+0000 mgr.smithi058.zagdlk (mgr.14164) 160 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:29.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:29 smithi187 conmon[36358]: cluster 2021-11-20T05:33:28.202286+0000 mgr.smithi058.zagdlk (mgr.14164) 160 : cluster 2021-11-20T05:33:29.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:29 smithi187 conmon[36358]: [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:30.511 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.506+0000 7f4a69176700 1 -- 172.21.15.58:0/470126066 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4a640f9800 msgr2=0x7f4a640fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:30.512 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.506+0000 7f4a69176700 1 --2- 172.21.15.58:0/470126066 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4a640f9800 0x7f4a640fbbf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f4a54004660 tx=0x7f4a54009b30).stop 2021-11-20T05:33:30.512 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.506+0000 7f4a69176700 1 -- 172.21.15.58:0/470126066 shutdown_connections 2021-11-20T05:33:30.512 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.506+0000 7f4a69176700 1 --2- 172.21.15.58:0/470126066 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4a640f9800 0x7f4a640fbbf0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:30.512 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.506+0000 7f4a69176700 1 --2- 172.21.15.58:0/470126066 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a640607f0 0x7f4a640f9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:33:30.512 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.506+0000 7f4a69176700 1 -- 172.21.15.58:0/470126066 >> 172.21.15.58:0/470126066 conn(0x7f4a640f0ca0 msgr2=0x7f4a640f30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:30.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.507+0000 7f4a69176700 1 -- 172.21.15.58:0/470126066 shutdown_connections 2021-11-20T05:33:30.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.507+0000 7f4a69176700 1 -- 172.21.15.58:0/470126066 wait complete. 2021-11-20T05:33:30.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.507+0000 7f4a69176700 1 Processor -- start 2021-11-20T05:33:30.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.507+0000 7f4a69176700 1 -- start start 2021-11-20T05:33:30.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.508+0000 7f4a69176700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a640607f0 0x7f4a64106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:30.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.508+0000 7f4a69176700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4a640f9800 0x7f4a641073e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:30.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.508+0000 7f4a69176700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a64107a20 con 0x7f4a640f9800 2021-11-20T05:33:30.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.508+0000 7f4a69176700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a641082d0 con 0x7f4a640607f0 2021-11-20T05:33:30.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.508+0000 7f4a637fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4a640f9800 0x7f4a641073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:30.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.508+0000 7f4a637fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4a640f9800 0x7f4a641073e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40314/0 (socket says 172.21.15.58:40314) 2021-11-20T05:33:30.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.508+0000 7f4a637fe700 1 -- 172.21.15.58:0/131002542 learned_addr learned my addr 172.21.15.58:0/131002542 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:30.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.508+0000 7f4a637fe700 1 -- 172.21.15.58:0/131002542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a640607f0 msgr2=0x7f4a64106ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:30.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.508+0000 7f4a63fff700 1 --2- 172.21.15.58:0/131002542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a640607f0 0x7f4a64106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:30.516 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.508+0000 7f4a637fe700 1 --2- 172.21.15.58:0/131002542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a640607f0 0x7f4a64106ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:30.516 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.508+0000 7f4a637fe700 1 -- 172.21.15.58:0/131002542 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a54005040 con 0x7f4a640f9800 2021-11-20T05:33:30.516 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.508+0000 7f4a637fe700 1 --2- 172.21.15.58:0/131002542 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4a640f9800 0x7f4a641073e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f4a54009f50 tx=0x7f4a54009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:30.516 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.508+0000 7f4a60ff9700 1 -- 172.21.15.58:0/131002542 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a54018070 con 0x7f4a640f9800 2021-11-20T05:33:30.517 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.508+0000 7f4a60ff9700 1 -- 172.21.15.58:0/131002542 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a54013410 con 0x7f4a640f9800 2021-11-20T05:33:30.517 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.508+0000 7f4a69176700 1 -- 172.21.15.58:0/131002542 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a64108550 con 0x7f4a640f9800 2021-11-20T05:33:30.517 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.509+0000 7f4a69176700 1 -- 172.21.15.58:0/131002542 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a64108ab0 con 0x7f4a640f9800 2021-11-20T05:33:30.518 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.509+0000 7f4a60ff9700 1 -- 172.21.15.58:0/131002542 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a5401c620 con 0x7f4a640f9800 2021-11-20T05:33:30.518 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.509+0000 7f4a69176700 1 -- 172.21.15.58:0/131002542 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4a48000fc0 con 0x7f4a640f9800 2021-11-20T05:33:30.518 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.510+0000 7f4a60ff9700 1 -- 172.21.15.58:0/131002542 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4a5401d420 con 0x7f4a640f9800 2021-11-20T05:33:30.519 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.510+0000 7f4a60ff9700 1 --2- 172.21.15.58:0/131002542 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4a5005c2b0 0x7f4a5005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:30.519 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.510+0000 7f4a60ff9700 1 -- 172.21.15.58:0/131002542 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f4a54077860 con 0x7f4a640f9800 2021-11-20T05:33:30.519 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.510+0000 7f4a63fff700 1 --2- 172.21.15.58:0/131002542 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4a5005c2b0 0x7f4a5005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:30.520 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.510+0000 7f4a60ff9700 1 --2- 172.21.15.58:0/131002542 >> [v2:172.21.15.187:6808/3483552299,v1:172.21.15.187:6809/3483552299] conn(0x7f4a50062040 0x7f4a500644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:30.520 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.510+0000 7f4a60ff9700 1 -- 172.21.15.58:0/131002542 --> [v2:172.21.15.187:6808/3483552299,v1:172.21.15.187:6809/3483552299] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a50064b20 con 0x7f4a50062040 2021-11-20T05:33:30.521 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.510+0000 7f4a60ff9700 1 -- 172.21.15.58:0/131002542 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a5407c220 con 0x7f4a640f9800 2021-11-20T05:33:30.521 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.511+0000 7f4a68975700 1 --2- 172.21.15.58:0/131002542 >> [v2:172.21.15.187:6808/3483552299,v1:172.21.15.187:6809/3483552299] conn(0x7f4a50062040 0x7f4a500644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:30.521 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.511+0000 7f4a63fff700 1 --2- 172.21.15.58:0/131002542 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4a5005c2b0 0x7f4a5005e770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f4a4c00a6f0 tx=0x7f4a4c008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:30.522 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.511+0000 7f4a68975700 1 --2- 172.21.15.58:0/131002542 >> [v2:172.21.15.187:6808/3483552299,v1:172.21.15.187:6809/3483552299] conn(0x7f4a50062040 0x7f4a500644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:30.522 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.512+0000 7f4a60ff9700 1 -- 172.21.15.58:0/131002542 <== osd.4 v2:172.21.15.187:6808/3483552299 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f4a50064b20 con 0x7f4a50062040 2021-11-20T05:33:30.525 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.526+0000 7f4a69176700 1 -- 172.21.15.58:0/131002542 --> [v2:172.21.15.187:6808/3483552299,v1:172.21.15.187:6809/3483552299] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4a480029d0 con 0x7f4a50062040 2021-11-20T05:33:30.526 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.527+0000 7f4a60ff9700 1 -- 172.21.15.58:0/131002542 <== osd.4 v2:172.21.15.187:6808/3483552299 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4a480029d0 con 0x7f4a50062040 2021-11-20T05:33:30.528 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.527+0000 7f4a69176700 1 -- 172.21.15.58:0/131002542 >> [v2:172.21.15.187:6808/3483552299,v1:172.21.15.187:6809/3483552299] conn(0x7f4a50062040 msgr2=0x7f4a500644a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:30.528 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.527+0000 7f4a69176700 1 --2- 172.21.15.58:0/131002542 >> [v2:172.21.15.187:6808/3483552299,v1:172.21.15.187:6809/3483552299] conn(0x7f4a50062040 0x7f4a500644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:30.528 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.527+0000 7f4a69176700 1 -- 172.21.15.58:0/131002542 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4a5005c2b0 msgr2=0x7f4a5005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:30.528 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.527+0000 7f4a69176700 1 --2- 172.21.15.58:0/131002542 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4a5005c2b0 0x7f4a5005e770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f4a4c00a6f0 tx=0x7f4a4c008040).stop 2021-11-20T05:33:30.529 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.527+0000 7f4a69176700 1 -- 172.21.15.58:0/131002542 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4a640f9800 msgr2=0x7f4a641073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:30.529 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.527+0000 7f4a69176700 1 --2- 172.21.15.58:0/131002542 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4a640f9800 0x7f4a641073e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f4a54009f50 tx=0x7f4a54009b30).stop 2021-11-20T05:33:30.529 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.527+0000 7f4a69176700 1 -- 172.21.15.58:0/131002542 shutdown_connections 2021-11-20T05:33:30.529 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.527+0000 7f4a69176700 1 --2- 172.21.15.58:0/131002542 >> [v2:172.21.15.187:6808/3483552299,v1:172.21.15.187:6809/3483552299] conn(0x7f4a50062040 0x7f4a500644a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:30.529 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.527+0000 7f4a69176700 1 --2- 172.21.15.58:0/131002542 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4a5005c2b0 0x7f4a5005e770 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:30.530 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.527+0000 7f4a69176700 1 --2- 172.21.15.58:0/131002542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a640607f0 0x7f4a64106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:30.530 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.527+0000 7f4a69176700 1 --2- 172.21.15.58:0/131002542 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4a640f9800 0x7f4a641073e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:30.530 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.527+0000 7f4a69176700 1 -- 172.21.15.58:0/131002542 >> 172.21.15.58:0/131002542 conn(0x7f4a640f0ca0 msgr2=0x7f4a640f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:30.530 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.527+0000 7f4a69176700 1 -- 172.21.15.58:0/131002542 shutdown_connections 2021-11-20T05:33:30.531 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:30.527+0000 7f4a69176700 1 -- 172.21.15.58:0/131002542 wait complete. 2021-11-20T05:33:30.950 INFO:teuthology.orchestra.run.smithi058.stdout:133143986187 2021-11-20T05:33:30.951 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2021-11-20T05:33:31.393 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:31.537 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:31 smithi058 conmon[33537]: cluster 2021-11-20T05:33:30.202761 2021-11-20T05:33:31.538 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:31 smithi058 conmon[33537]: +0000 mgr.smithi058.zagdlk (mgr.14164) 161 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:31.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:31 smithi187 conmon[36358]: cluster 2021-11-20T05:33:30.202761+0000 mgr.smithi058.zagdlk (mgr.14164) 161 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:32.576 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.574+0000 7ff21a069700 1 -- 172.21.15.58:0/879835967 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff2140f6af0 msgr2=0x7ff2140f6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:32.577 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.574+0000 7ff21a069700 1 --2- 172.21.15.58:0/879835967 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff2140f6af0 0x7ff2140f6f10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7ff210004660 tx=0x7ff210009b30).stop 2021-11-20T05:33:32.577 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.575+0000 7ff21a069700 1 -- 172.21.15.58:0/879835967 shutdown_connections 2021-11-20T05:33:32.577 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.575+0000 7ff21a069700 1 --2- 172.21.15.58:0/879835967 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2140f7490 0x7ff2140f4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:32.578 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.575+0000 7ff21a069700 1 --2- 172.21.15.58:0/879835967 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff2140f6af0 0x7ff2140f6f10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:32.578 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.575+0000 7ff21a069700 1 -- 172.21.15.58:0/879835967 >> 172.21.15.58:0/879835967 conn(0x7ff2140f0400 msgr2=0x7ff2140f2820 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:32.578 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.575+0000 7ff21a069700 1 -- 172.21.15.58:0/879835967 shutdown_connections 2021-11-20T05:33:32.579 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.575+0000 7ff21a069700 1 -- 172.21.15.58:0/879835967 wait complete. 2021-11-20T05:33:32.579 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.575+0000 7ff21a069700 1 Processor -- start 2021-11-20T05:33:32.579 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.575+0000 7ff21a069700 1 -- start start 2021-11-20T05:33:32.579 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.575+0000 7ff21a069700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2140f6af0 0x7ff2141090e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:32.580 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.575+0000 7ff21a069700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff2140f7490 0x7ff2141095d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:32.580 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.575+0000 7ff21a069700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff214109b80 con 0x7ff2140f7490 2021-11-20T05:33:32.580 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.575+0000 7ff21a069700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff214109cc0 con 0x7ff2140f6af0 2021-11-20T05:33:32.580 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.575+0000 7ff218866700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff2140f7490 0x7ff2141095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:32.580 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.576+0000 7ff218866700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff2140f7490 0x7ff2141095d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40326/0 (socket says 172.21.15.58:40326) 2021-11-20T05:33:32.581 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.576+0000 7ff218866700 1 -- 172.21.15.58:0/2125437138 learned_addr learned my addr 172.21.15.58:0/2125437138 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:32.581 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.576+0000 7ff219067700 1 --2- 172.21.15.58:0/2125437138 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2140f6af0 0x7ff2141090e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:32.581 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.576+0000 7ff218866700 1 -- 172.21.15.58:0/2125437138 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2140f6af0 msgr2=0x7ff2141090e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:32.581 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.576+0000 7ff218866700 1 --2- 172.21.15.58:0/2125437138 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2140f6af0 0x7ff2141090e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:32.582 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.576+0000 7ff218866700 1 -- 172.21.15.58:0/2125437138 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff210005040 con 0x7ff2140f7490 2021-11-20T05:33:32.582 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.576+0000 7ff218866700 1 --2- 172.21.15.58:0/2125437138 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff2140f7490 0x7ff2141095d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7ff20400c530 tx=0x7ff204007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:32.582 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.576+0000 7ff209ffb700 1 -- 172.21.15.58:0/2125437138 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff20400f650 con 0x7ff2140f7490 2021-11-20T05:33:32.582 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.576+0000 7ff209ffb700 1 -- 172.21.15.58:0/2125437138 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff20400fc60 con 0x7ff2140f7490 2021-11-20T05:33:32.583 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.576+0000 7ff21a069700 1 -- 172.21.15.58:0/2125437138 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff21410a660 con 0x7ff2140f7490 2021-11-20T05:33:32.583 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.576+0000 7ff209ffb700 1 -- 172.21.15.58:0/2125437138 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff204012c30 con 0x7ff2140f7490 2021-11-20T05:33:32.583 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.576+0000 7ff21a069700 1 -- 172.21.15.58:0/2125437138 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff21410ac20 con 0x7ff2140f7490 2021-11-20T05:33:32.583 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.577+0000 7ff21a069700 1 -- 172.21.15.58:0/2125437138 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff1f4000fc0 con 0x7ff2140f7490 2021-11-20T05:33:32.585 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.578+0000 7ff209ffb700 1 -- 172.21.15.58:0/2125437138 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff204010070 con 0x7ff2140f7490 2021-11-20T05:33:32.585 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.578+0000 7ff209ffb700 1 --2- 172.21.15.58:0/2125437138 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff20005c2b0 0x7ff20005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:32.585 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.578+0000 7ff209ffb700 1 -- 172.21.15.58:0/2125437138 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7ff2040756b0 con 0x7ff2140f7490 2021-11-20T05:33:32.586 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.578+0000 7ff219067700 1 --2- 172.21.15.58:0/2125437138 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff20005c2b0 0x7ff20005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:32.586 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.578+0000 7ff209ffb700 1 --2- 172.21.15.58:0/2125437138 >> [v2:172.21.15.187:6816/566077514,v1:172.21.15.187:6817/566077514] conn(0x7ff200062040 0x7ff2000644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:32.586 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.578+0000 7ff209ffb700 1 -- 172.21.15.58:0/2125437138 --> [v2:172.21.15.187:6816/566077514,v1:172.21.15.187:6817/566077514] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff200064b20 con 0x7ff200062040 2021-11-20T05:33:32.586 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.578+0000 7ff209ffb700 1 -- 172.21.15.58:0/2125437138 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff204077d20 con 0x7ff2140f7490 2021-11-20T05:33:32.587 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.578+0000 7ff219868700 1 --2- 172.21.15.58:0/2125437138 >> [v2:172.21.15.187:6816/566077514,v1:172.21.15.187:6817/566077514] conn(0x7ff200062040 0x7ff2000644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:32.587 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.578+0000 7ff219067700 1 --2- 172.21.15.58:0/2125437138 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff20005c2b0 0x7ff20005e770 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7ff210004630 tx=0x7ff210009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:32.587 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.579+0000 7ff219868700 1 --2- 172.21.15.58:0/2125437138 >> [v2:172.21.15.187:6816/566077514,v1:172.21.15.187:6817/566077514] conn(0x7ff200062040 0x7ff2000644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:32.587 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.580+0000 7ff209ffb700 1 -- 172.21.15.58:0/2125437138 <== osd.5 v2:172.21.15.187:6816/566077514 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7ff200064b20 con 0x7ff200062040 2021-11-20T05:33:32.593 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.594+0000 7ff21a069700 1 -- 172.21.15.58:0/2125437138 --> [v2:172.21.15.187:6816/566077514,v1:172.21.15.187:6817/566077514] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff1f40029d0 con 0x7ff200062040 2021-11-20T05:33:32.594 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.595+0000 7ff209ffb700 1 -- 172.21.15.58:0/2125437138 <== osd.5 v2:172.21.15.187:6816/566077514 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff1f40029d0 con 0x7ff200062040 2021-11-20T05:33:32.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.595+0000 7ff21a069700 1 -- 172.21.15.58:0/2125437138 >> [v2:172.21.15.187:6816/566077514,v1:172.21.15.187:6817/566077514] conn(0x7ff200062040 msgr2=0x7ff2000644a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:32.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.595+0000 7ff21a069700 1 --2- 172.21.15.58:0/2125437138 >> [v2:172.21.15.187:6816/566077514,v1:172.21.15.187:6817/566077514] conn(0x7ff200062040 0x7ff2000644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:32.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.595+0000 7ff21a069700 1 -- 172.21.15.58:0/2125437138 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff20005c2b0 msgr2=0x7ff20005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:32.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.595+0000 7ff21a069700 1 --2- 172.21.15.58:0/2125437138 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff20005c2b0 0x7ff20005e770 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7ff210004630 tx=0x7ff210009b30).stop 2021-11-20T05:33:32.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.595+0000 7ff21a069700 1 -- 172.21.15.58:0/2125437138 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff2140f7490 msgr2=0x7ff2141095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:32.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.595+0000 7ff21a069700 1 --2- 172.21.15.58:0/2125437138 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff2140f7490 0x7ff2141095d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7ff20400c530 tx=0x7ff204007e20).stop 2021-11-20T05:33:32.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.595+0000 7ff21a069700 1 -- 172.21.15.58:0/2125437138 shutdown_connections 2021-11-20T05:33:32.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.595+0000 7ff21a069700 1 --2- 172.21.15.58:0/2125437138 >> [v2:172.21.15.187:6816/566077514,v1:172.21.15.187:6817/566077514] conn(0x7ff200062040 0x7ff2000644a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:32.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.595+0000 7ff21a069700 1 --2- 172.21.15.58:0/2125437138 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff20005c2b0 0x7ff20005e770 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:32.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.595+0000 7ff21a069700 1 --2- 172.21.15.58:0/2125437138 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2140f6af0 0x7ff2141090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:32.598 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.595+0000 7ff21a069700 1 --2- 172.21.15.58:0/2125437138 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff2140f7490 0x7ff2141095d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:32.598 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.595+0000 7ff21a069700 1 -- 172.21.15.58:0/2125437138 >> 172.21.15.58:0/2125437138 conn(0x7ff2140f0400 msgr2=0x7ff2140fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:32.598 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.595+0000 7ff21a069700 1 -- 172.21.15.58:0/2125437138 shutdown_connections 2021-11-20T05:33:32.599 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:32.595+0000 7ff21a069700 1 -- 172.21.15.58:0/2125437138 wait complete. 2021-11-20T05:33:33.005 INFO:teuthology.orchestra.run.smithi058.stdout:154618822664 2021-11-20T05:33:33.006 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-11-20T05:33:33.451 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:33.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:33 smithi058 conmon[33537]: cluster 2021-11-20T05:33:32.203054+0000 mgr.smithi058.zagdlk (mgr.14164) 162 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:33.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:33 smithi058 conmon[33537]: 2021-11-20T05:33:33.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:33 smithi187 conmon[36358]: cluster 2021-11-20T05:33:32.203054+0000 mgr.smithi058.zagdlk (mgr.14164) 162 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:34.615 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.612+0000 7f18434c5700 1 -- 172.21.15.58:0/3217277703 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f183c060870 msgr2=0x7f183c0f93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:34.616 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.612+0000 7f18434c5700 1 --2- 172.21.15.58:0/3217277703 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f183c060870 0x7f183c0f93a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f1834004660 tx=0x7f1834009b30).stop 2021-11-20T05:33:34.616 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.612+0000 7f18434c5700 1 -- 172.21.15.58:0/3217277703 shutdown_connections 2021-11-20T05:33:34.616 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.612+0000 7f18434c5700 1 --2- 172.21.15.58:0/3217277703 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f183c0f9920 0x7f183c0fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:33:34.617 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.612+0000 7f18434c5700 1 --2- 172.21.15.58:0/3217277703 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f183c060870 0x7f183c0f93a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:34.617 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.612+0000 7f18434c5700 1 -- 172.21.15.58:0/3217277703 >> 172.21.15.58:0/3217277703 conn(0x7f183c0f0e20 msgr2=0x7f183c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:34.617 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.612+0000 7f18434c5700 1 -- 172.21.15.58:0/3217277703 shutdown_connections 2021-11-20T05:33:34.618 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.612+0000 7f18434c5700 1 -- 172.21.15.58:0/3217277703 wait complete. 2021-11-20T05:33:34.618 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.612+0000 7f18434c5700 1 Processor -- start 2021-11-20T05:33:34.618 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.612+0000 7f18434c5700 1 -- start start 2021-11-20T05:33:34.618 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.613+0000 7f18434c5700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f183c060870 0x7f183c0f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:34.619 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.613+0000 7f18434c5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f183c0f9920 0x7f183c0f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:34.619 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.613+0000 7f18434c5700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c0f5890 con 0x7f183c060870 2021-11-20T05:33:34.619 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.613+0000 7f18434c5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c0f59d0 con 0x7f183c0f9920 2021-11-20T05:33:34.619 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.613+0000 7f1841261700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f183c060870 0x7f183c0f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:34.620 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.613+0000 7f1841261700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f183c060870 0x7f183c0f6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40336/0 (socket says 172.21.15.58:40336) 2021-11-20T05:33:34.620 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.613+0000 7f1841261700 1 -- 172.21.15.58:0/905964531 learned_addr learned my addr 172.21.15.58:0/905964531 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:34.620 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.613+0000 7f1841261700 1 -- 172.21.15.58:0/905964531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f183c0f9920 msgr2=0x7f183c0f52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:34.620 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.613+0000 7f1841261700 1 --2- 172.21.15.58:0/905964531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f183c0f9920 0x7f183c0f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:33:34.621 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.613+0000 7f1841261700 1 -- 172.21.15.58:0/905964531 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1834005040 con 0x7f183c060870 2021-11-20T05:33:34.621 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.613+0000 7f1841261700 1 --2- 172.21.15.58:0/905964531 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f183c060870 0x7f183c0f6e20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f1834009830 tx=0x7f1834009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:34.621 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.613+0000 7f1839ffb700 1 -- 172.21.15.58:0/905964531 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1834018070 con 0x7f183c060870 2021-11-20T05:33:34.621 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.613+0000 7f1839ffb700 1 -- 172.21.15.58:0/905964531 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1834013410 con 0x7f183c060870 2021-11-20T05:33:34.621 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.613+0000 7f18434c5700 1 -- 172.21.15.58:0/905964531 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183c0f5c50 con 0x7f183c060870 2021-11-20T05:33:34.622 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.613+0000 7f1839ffb700 1 -- 172.21.15.58:0/905964531 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f183401c620 con 0x7f183c060870 2021-11-20T05:33:34.622 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.613+0000 7f18434c5700 1 -- 172.21.15.58:0/905964531 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f183c10f0d0 con 0x7f183c060870 2021-11-20T05:33:34.622 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.614+0000 7f18434c5700 1 -- 172.21.15.58:0/905964531 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f183c045ad0 con 0x7f183c060870 2021-11-20T05:33:34.622 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.615+0000 7f1839ffb700 1 -- 172.21.15.58:0/905964531 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f183401c8e0 con 0x7f183c060870 2021-11-20T05:33:34.623 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.615+0000 7f1839ffb700 1 --2- 172.21.15.58:0/905964531 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f182405c2a0 0x7f182405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:34.623 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.615+0000 7f1839ffb700 1 -- 172.21.15.58:0/905964531 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f1834077a60 con 0x7f183c060870 2021-11-20T05:33:34.623 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.615+0000 7f1840a60700 1 --2- 172.21.15.58:0/905964531 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f182405c2a0 0x7f182405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:34.623 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.616+0000 7f1840a60700 1 --2- 172.21.15.58:0/905964531 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f182405c2a0 0x7f182405e760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f183c060640 tx=0x7f1830006b50).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:34.624 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.619+0000 7f1839ffb700 1 -- 172.21.15.58:0/905964531 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f183404aa00 con 0x7f183c060870 2021-11-20T05:33:34.780 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.780+0000 7f18434c5700 1 -- 172.21.15.58:0/905964531 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f183c027a00 con 0x7f183c060870 2021-11-20T05:33:34.780 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.780+0000 7f1839ffb700 1 -- 172.21.15.58:0/905964531 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f183404e020 con 0x7f183c060870 2021-11-20T05:33:34.781 INFO:teuthology.orchestra.run.smithi058.stdout:38654705689 2021-11-20T05:33:34.782 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.782+0000 7f18434c5700 1 -- 172.21.15.58:0/905964531 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f182405c2a0 msgr2=0x7f182405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:34.782 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.782+0000 7f18434c5700 1 --2- 172.21.15.58:0/905964531 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f182405c2a0 0x7f182405e760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f183c060640 tx=0x7f1830006b50).stop 2021-11-20T05:33:34.782 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.782+0000 7f18434c5700 1 -- 172.21.15.58:0/905964531 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f183c060870 msgr2=0x7f183c0f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:34.783 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.782+0000 7f18434c5700 1 --2- 172.21.15.58:0/905964531 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f183c060870 0x7f183c0f6e20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f1834009830 tx=0x7f1834009b30).stop 2021-11-20T05:33:34.783 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.782+0000 7f18434c5700 1 -- 172.21.15.58:0/905964531 shutdown_connections 2021-11-20T05:33:34.783 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.782+0000 7f18434c5700 1 --2- 172.21.15.58:0/905964531 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f182405c2a0 0x7f182405e760 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:34.783 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.782+0000 7f18434c5700 1 --2- 172.21.15.58:0/905964531 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f183c060870 0x7f183c0f6e20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:34.784 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.782+0000 7f18434c5700 1 --2- 172.21.15.58:0/905964531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f183c0f9920 0x7f183c0f52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:33:34.784 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.782+0000 7f18434c5700 1 -- 172.21.15.58:0/905964531 >> 172.21.15.58:0/905964531 conn(0x7f183c0f0e20 msgr2=0x7f183c0f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:34.784 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.782+0000 7f18434c5700 1 -- 172.21.15.58:0/905964531 shutdown_connections 2021-11-20T05:33:34.784 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:34.782+0000 7f18434c5700 1 -- 172.21.15.58:0/905964531 wait complete. 2021-11-20T05:33:35.067 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705687 got 38654705689 for osd.0 2021-11-20T05:33:35.068 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.1 2021-11-20T05:33:35.517 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:35.656 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:35 smithi058 conmon[33537]: cluster 2021-11-20T05:33:34.203374+0000 mgr.smithi058.zagdlk (mgr.14164) 163 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:35.657 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:35 smithi058 conmon[33537]: audit 2021-11-20T05:33:34.781477+0000 mon.smithi058 (mon.0) 436 : audit [DBG] from='client.? 172.21.15.58:0/905964531' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-11-20T05:33:35.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:35 smithi187 conmon[36358]: cluster 2021-11-20T05:33:34.203374+0000 mgr.smithi058.zagdlk (mgr.14164) 163 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:35.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:35 smithi187 conmon[36358]: audit 2021-11-20T05:33:34.781477+0000 mon.smithi058 (mon.0) 436 : audit [DBG] from='client.? 172.21.15.58:0/905964531' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-11-20T05:33:36.656 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.656+0000 7f8b23a8d700 1 -- 172.21.15.58:0/4028802409 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b1c0f6980 msgr2=0x7f8b1c0f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:36.657 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.656+0000 7f8b23a8d700 1 --2- 172.21.15.58:0/4028802409 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b1c0f6980 0x7f8b1c0f6de0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f8b18004660 tx=0x7f8b18009b30).stop 2021-11-20T05:33:36.657 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.656+0000 7f8b23a8d700 1 -- 172.21.15.58:0/4028802409 shutdown_connections 2021-11-20T05:33:36.662 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.656+0000 7f8b23a8d700 1 --2- 172.21.15.58:0/4028802409 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b1c0f6980 0x7f8b1c0f6de0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:36.663 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.656+0000 7f8b23a8d700 1 --2- 172.21.15.58:0/4028802409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b1c0f56e0 0x7f8b1c0f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:33:36.663 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.656+0000 7f8b23a8d700 1 -- 172.21.15.58:0/4028802409 >> 172.21.15.58:0/4028802409 conn(0x7f8b1c0f0e20 msgr2=0x7f8b1c0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:36.663 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.656+0000 7f8b23a8d700 1 -- 172.21.15.58:0/4028802409 shutdown_connections 2021-11-20T05:33:36.663 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.656+0000 7f8b23a8d700 1 -- 172.21.15.58:0/4028802409 wait complete. 2021-11-20T05:33:36.664 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.657+0000 7f8b23a8d700 1 Processor -- start 2021-11-20T05:33:36.664 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.657+0000 7f8b23a8d700 1 -- start start 2021-11-20T05:33:36.666 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.657+0000 7f8b23a8d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b1c0f56e0 0x7f8b1c105ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:36.667 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.657+0000 7f8b23a8d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b1c0f6980 0x7f8b1c1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:36.667 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.657+0000 7f8b23a8d700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c106b20 con 0x7f8b1c0f56e0 2021-11-20T05:33:36.668 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.657+0000 7f8b23a8d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c1073d0 con 0x7f8b1c0f6980 2021-11-20T05:33:36.668 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.657+0000 7f8b21829700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b1c0f56e0 0x7f8b1c105ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:36.668 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.657+0000 7f8b21829700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b1c0f56e0 0x7f8b1c105ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40346/0 (socket says 172.21.15.58:40346) 2021-11-20T05:33:36.669 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.657+0000 7f8b21829700 1 -- 172.21.15.58:0/4290986913 learned_addr learned my addr 172.21.15.58:0/4290986913 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:36.669 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.657+0000 7f8b21028700 1 --2- 172.21.15.58:0/4290986913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b1c0f6980 0x7f8b1c1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:36.669 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.657+0000 7f8b21829700 1 -- 172.21.15.58:0/4290986913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b1c0f6980 msgr2=0x7f8b1c1064e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:36.669 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.657+0000 7f8b21829700 1 --2- 172.21.15.58:0/4290986913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b1c0f6980 0x7f8b1c1064e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:36.670 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.657+0000 7f8b21829700 1 -- 172.21.15.58:0/4290986913 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b18005040 con 0x7f8b1c0f56e0 2021-11-20T05:33:36.670 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.657+0000 7f8b21829700 1 --2- 172.21.15.58:0/4290986913 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b1c0f56e0 0x7f8b1c105ff0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f8b100093f0 tx=0x7f8b10006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:36.670 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.658+0000 7f8b0e7fc700 1 -- 172.21.15.58:0/4290986913 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b10009ea0 con 0x7f8b1c0f56e0 2021-11-20T05:33:36.670 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.658+0000 7f8b23a8d700 1 -- 172.21.15.58:0/4290986913 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b1c107650 con 0x7f8b1c0f56e0 2021-11-20T05:33:36.671 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.658+0000 7f8b0e7fc700 1 -- 172.21.15.58:0/4290986913 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b10013af0 con 0x7f8b1c0f56e0 2021-11-20T05:33:36.671 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.658+0000 7f8b0e7fc700 1 -- 172.21.15.58:0/4290986913 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b1000dad0 con 0x7f8b1c0f56e0 2021-11-20T05:33:36.671 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.658+0000 7f8b23a8d700 1 -- 172.21.15.58:0/4290986913 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b1c107b30 con 0x7f8b1c0f56e0 2021-11-20T05:33:36.671 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.659+0000 7f8b23a8d700 1 -- 172.21.15.58:0/4290986913 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b1c045ad0 con 0x7f8b1c0f56e0 2021-11-20T05:33:36.671 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.659+0000 7f8b0e7fc700 1 -- 172.21.15.58:0/4290986913 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8b1001a060 con 0x7f8b1c0f56e0 2021-11-20T05:33:36.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.659+0000 7f8b0e7fc700 1 --2- 172.21.15.58:0/4290986913 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b0805c2a0 0x7f8b0805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:36.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.659+0000 7f8b0e7fc700 1 -- 172.21.15.58:0/4290986913 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8b10075ba0 con 0x7f8b1c0f56e0 2021-11-20T05:33:36.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.660+0000 7f8b21028700 1 --2- 172.21.15.58:0/4290986913 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b0805c2a0 0x7f8b0805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:36.673 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.660+0000 7f8b21028700 1 --2- 172.21.15.58:0/4290986913 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b0805c2a0 0x7f8b0805e760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f8b18013580 tx=0x7f8b18009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:36.673 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.664+0000 7f8b0e7fc700 1 -- 172.21.15.58:0/4290986913 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b10048b40 con 0x7f8b1c0f56e0 2021-11-20T05:33:36.821 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.821+0000 7f8b23a8d700 1 -- 172.21.15.58:0/4290986913 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8b1c1081d0 con 0x7f8b1c0f56e0 2021-11-20T05:33:36.821 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.821+0000 7f8b0e7fc700 1 -- 172.21.15.58:0/4290986913 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8b1004c160 con 0x7f8b1c0f56e0 2021-11-20T05:33:36.821 INFO:teuthology.orchestra.run.smithi058.stdout:60129542167 2021-11-20T05:33:36.823 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.823+0000 7f8b23a8d700 1 -- 172.21.15.58:0/4290986913 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b0805c2a0 msgr2=0x7f8b0805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:36.823 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.823+0000 7f8b23a8d700 1 --2- 172.21.15.58:0/4290986913 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b0805c2a0 0x7f8b0805e760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f8b18013580 tx=0x7f8b18009b30).stop 2021-11-20T05:33:36.823 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.823+0000 7f8b23a8d700 1 -- 172.21.15.58:0/4290986913 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b1c0f56e0 msgr2=0x7f8b1c105ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:36.824 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.823+0000 7f8b23a8d700 1 --2- 172.21.15.58:0/4290986913 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b1c0f56e0 0x7f8b1c105ff0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f8b100093f0 tx=0x7f8b10006ec0).stop 2021-11-20T05:33:36.824 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.823+0000 7f8b23a8d700 1 -- 172.21.15.58:0/4290986913 shutdown_connections 2021-11-20T05:33:36.824 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.823+0000 7f8b23a8d700 1 --2- 172.21.15.58:0/4290986913 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b0805c2a0 0x7f8b0805e760 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:36.824 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.823+0000 7f8b23a8d700 1 --2- 172.21.15.58:0/4290986913 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b1c0f56e0 0x7f8b1c105ff0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:36.825 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.823+0000 7f8b23a8d700 1 --2- 172.21.15.58:0/4290986913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b1c0f6980 0x7f8b1c1064e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:36.825 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.823+0000 7f8b23a8d700 1 -- 172.21.15.58:0/4290986913 >> 172.21.15.58:0/4290986913 conn(0x7f8b1c0f0e20 msgr2=0x7f8b1c0fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:36.825 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.823+0000 7f8b23a8d700 1 -- 172.21.15.58:0/4290986913 shutdown_connections 2021-11-20T05:33:36.826 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:36.823+0000 7f8b23a8d700 1 -- 172.21.15.58:0/4290986913 wait complete. 2021-11-20T05:33:37.230 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542165 got 60129542167 for osd.1 2021-11-20T05:33:37.231 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.2 2021-11-20T05:33:37.679 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:37.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:37 smithi187 conmon[36358]: cluster 2021-11-20T05:33:36.203765+0000 mgr.smithi058.zagdlk (mgr.14164) 164 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:37.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:37 smithi187 conmon[36358]: 2021-11-20T05:33:37.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:37 smithi187 conmon[36358]: audit 2021-11-20T05:33:36.822396+0000 mon.smithi058 (mon.0) 437 : audit [DBG] from='client.? 172.21.15.58:0/4290986913' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-11-20T05:33:37.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:37 smithi058 conmon[33537]: cluster 2021-11-20T05:33:36.203765+0000 mgr.smithi058.zagdlk (mgr.14164) 164 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:37.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:37 smithi058 conmon[33537]: audit 2021-11-20T05:33:36.822396+0000 mon.smithi058 (mon.0) 437 : audit [DBG] from='client.? 172.21.15.58:0/4290986913' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-11-20T05:33:38.830 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.829+0000 7f0b23f29700 1 -- 172.21.15.58:0/429196855 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1c060870 msgr2=0x7f0b1c0f93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:38.831 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.829+0000 7f0b23f29700 1 --2- 172.21.15.58:0/429196855 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1c060870 0x7f0b1c0f93a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f0b140099e0 tx=0x7f0b14004d10).stop 2021-11-20T05:33:38.831 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.829+0000 7f0b23f29700 1 -- 172.21.15.58:0/429196855 shutdown_connections 2021-11-20T05:33:38.831 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.829+0000 7f0b23f29700 1 --2- 172.21.15.58:0/429196855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b1c0f9920 0x7f0b1c0fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:38.832 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.829+0000 7f0b23f29700 1 --2- 172.21.15.58:0/429196855 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1c060870 0x7f0b1c0f93a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:38.832 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.829+0000 7f0b23f29700 1 -- 172.21.15.58:0/429196855 >> 172.21.15.58:0/429196855 conn(0x7f0b1c0f0e20 msgr2=0x7f0b1c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:38.832 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.829+0000 7f0b23f29700 1 -- 172.21.15.58:0/429196855 shutdown_connections 2021-11-20T05:33:38.832 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.829+0000 7f0b23f29700 1 -- 172.21.15.58:0/429196855 wait complete. 2021-11-20T05:33:38.832 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.829+0000 7f0b23f29700 1 Processor -- start 2021-11-20T05:33:38.833 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.830+0000 7f0b23f29700 1 -- start start 2021-11-20T05:33:38.833 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.830+0000 7f0b23f29700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1c060870 0x7f0b1c1071b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:38.833 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.830+0000 7f0b23f29700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b1c0f9920 0x7f0b1c1076a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:38.833 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.830+0000 7f0b23f29700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c107c50 con 0x7f0b1c060870 2021-11-20T05:33:38.834 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.830+0000 7f0b23f29700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c107d90 con 0x7f0b1c0f9920 2021-11-20T05:33:38.834 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.830+0000 7f0b21cc5700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1c060870 0x7f0b1c1071b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:38.834 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.830+0000 7f0b21cc5700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1c060870 0x7f0b1c1071b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40356/0 (socket says 172.21.15.58:40356) 2021-11-20T05:33:38.834 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.830+0000 7f0b21cc5700 1 -- 172.21.15.58:0/277762731 learned_addr learned my addr 172.21.15.58:0/277762731 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:38.835 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.830+0000 7f0b214c4700 1 --2- 172.21.15.58:0/277762731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b1c0f9920 0x7f0b1c1076a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:38.835 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.830+0000 7f0b21cc5700 1 -- 172.21.15.58:0/277762731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b1c0f9920 msgr2=0x7f0b1c1076a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:38.835 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.830+0000 7f0b21cc5700 1 --2- 172.21.15.58:0/277762731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b1c0f9920 0x7f0b1c1076a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:38.835 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.830+0000 7f0b21cc5700 1 -- 172.21.15.58:0/277762731 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b14009640 con 0x7f0b1c060870 2021-11-20T05:33:38.835 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.830+0000 7f0b21cc5700 1 --2- 172.21.15.58:0/277762731 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1c060870 0x7f0b1c1071b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f0b14004b70 tx=0x7f0b14004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:38.836 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.830+0000 7f0b127fc700 1 -- 172.21.15.58:0/277762731 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b1400e070 con 0x7f0b1c060870 2021-11-20T05:33:38.836 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.830+0000 7f0b23f29700 1 -- 172.21.15.58:0/277762731 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b1c1086d0 con 0x7f0b1c060870 2021-11-20T05:33:38.836 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.831+0000 7f0b127fc700 1 -- 172.21.15.58:0/277762731 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b14007dc0 con 0x7f0b1c060870 2021-11-20T05:33:38.836 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.831+0000 7f0b127fc700 1 -- 172.21.15.58:0/277762731 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b14018070 con 0x7f0b1c060870 2021-11-20T05:33:38.837 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.831+0000 7f0b23f29700 1 -- 172.21.15.58:0/277762731 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b1c108bb0 con 0x7f0b1c060870 2021-11-20T05:33:38.838 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.831+0000 7f0b23f29700 1 -- 172.21.15.58:0/277762731 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b1c045ad0 con 0x7f0b1c060870 2021-11-20T05:33:38.838 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.832+0000 7f0b127fc700 1 -- 172.21.15.58:0/277762731 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0b14024060 con 0x7f0b1c060870 2021-11-20T05:33:38.838 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.832+0000 7f0b127fc700 1 --2- 172.21.15.58:0/277762731 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0b0805c2a0 0x7f0b0805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:38.839 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.832+0000 7f0b127fc700 1 -- 172.21.15.58:0/277762731 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f0b140772d0 con 0x7f0b1c060870 2021-11-20T05:33:38.839 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.832+0000 7f0b214c4700 1 --2- 172.21.15.58:0/277762731 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0b0805c2a0 0x7f0b0805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:38.839 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.833+0000 7f0b214c4700 1 --2- 172.21.15.58:0/277762731 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0b0805c2a0 0x7f0b0805e760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f0b1c060640 tx=0x7f0b0c009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:38.839 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.836+0000 7f0b127fc700 1 -- 172.21.15.58:0/277762731 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b1404a350 con 0x7f0b1c060870 2021-11-20T05:33:38.997 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.997+0000 7f0b23f29700 1 -- 172.21.15.58:0/277762731 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f0b1c027a00 con 0x7f0b1c060870 2021-11-20T05:33:38.997 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.997+0000 7f0b127fc700 1 -- 172.21.15.58:0/277762731 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0b1404d970 con 0x7f0b1c060870 2021-11-20T05:33:38.998 INFO:teuthology.orchestra.run.smithi058.stdout:81604378642 2021-11-20T05:33:38.999 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.999+0000 7f0b23f29700 1 -- 172.21.15.58:0/277762731 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0b0805c2a0 msgr2=0x7f0b0805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:38.999 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.999+0000 7f0b23f29700 1 --2- 172.21.15.58:0/277762731 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0b0805c2a0 0x7f0b0805e760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f0b1c060640 tx=0x7f0b0c009be0).stop 2021-11-20T05:33:39.000 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.999+0000 7f0b23f29700 1 -- 172.21.15.58:0/277762731 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1c060870 msgr2=0x7f0b1c1071b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:39.000 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.999+0000 7f0b23f29700 1 --2- 172.21.15.58:0/277762731 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1c060870 0x7f0b1c1071b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f0b14004b70 tx=0x7f0b14004d10).stop 2021-11-20T05:33:39.000 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.999+0000 7f0b23f29700 1 -- 172.21.15.58:0/277762731 shutdown_connections 2021-11-20T05:33:39.000 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.999+0000 7f0b23f29700 1 --2- 172.21.15.58:0/277762731 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0b0805c2a0 0x7f0b0805e760 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:39.001 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.999+0000 7f0b23f29700 1 --2- 172.21.15.58:0/277762731 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b1c060870 0x7f0b1c1071b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:39.001 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.999+0000 7f0b23f29700 1 --2- 172.21.15.58:0/277762731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b1c0f9920 0x7f0b1c1076a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:39.001 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.999+0000 7f0b23f29700 1 -- 172.21.15.58:0/277762731 >> 172.21.15.58:0/277762731 conn(0x7f0b1c0f0e20 msgr2=0x7f0b1c0f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:39.001 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.999+0000 7f0b23f29700 1 -- 172.21.15.58:0/277762731 shutdown_connections 2021-11-20T05:33:39.002 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:38.999+0000 7f0b23f29700 1 -- 172.21.15.58:0/277762731 wait complete. 2021-11-20T05:33:39.581 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:39 smithi058 conmon[33537]: cluster 2021-11-20T05:33:38.204089+0000 mgr.smithi058.zagdlk (mgr.14164) 165 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:39.581 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:39 smithi058 conmon[33537]: audit 2021-11-20T05:33:38.998254+0000 mon.smithi058 (mon.0) 438 : audit [DBG] from='client.? 172.21.15.58:0/277762731' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-11-20T05:33:39.583 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378639 got 81604378642 for osd.2 2021-11-20T05:33:39.583 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.3 2021-11-20T05:33:39.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:39 smithi187 conmon[36358]: cluster 2021-11-20T05:33:38.204089+0000 mgr.smithi058.zagdlk (mgr.14164) 165 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:39.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:39 smithi187 conmon[36358]: audit 2021-11-20T05:33:38.998254 2021-11-20T05:33:39.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:39 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 438 : audit [DBG] from='client.? 172.21.15.58:0/277762731' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-11-20T05:33:40.031 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:41.318 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.316+0000 7fa70c863700 1 -- 172.21.15.58:0/2950908792 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa708060870 msgr2=0x7fa7080f93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:41.318 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.316+0000 7fa70c863700 1 --2- 172.21.15.58:0/2950908792 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa708060870 0x7fa7080f93c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fa7000099e0 tx=0x7fa700004d10).stop 2021-11-20T05:33:41.318 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.316+0000 7fa70c863700 1 -- 172.21.15.58:0/2950908792 shutdown_connections 2021-11-20T05:33:41.318 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.316+0000 7fa70c863700 1 --2- 172.21.15.58:0/2950908792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7080f9940 0x7fa7080fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:41.319 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.316+0000 7fa70c863700 1 --2- 172.21.15.58:0/2950908792 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa708060870 0x7fa7080f93c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:41.319 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.316+0000 7fa70c863700 1 -- 172.21.15.58:0/2950908792 >> 172.21.15.58:0/2950908792 conn(0x7fa7080f0e20 msgr2=0x7fa7080f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:41.319 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.316+0000 7fa70c863700 1 -- 172.21.15.58:0/2950908792 shutdown_connections 2021-11-20T05:33:41.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.316+0000 7fa70c863700 1 -- 172.21.15.58:0/2950908792 wait complete. 2021-11-20T05:33:41.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.316+0000 7fa70c863700 1 Processor -- start 2021-11-20T05:33:41.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.317+0000 7fa70c863700 1 -- start start 2021-11-20T05:33:41.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.317+0000 7fa70c863700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa708060870 0x7fa708107190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:41.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.317+0000 7fa70c863700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7080f9940 0x7fa708107680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:41.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.317+0000 7fa70c863700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa708107c30 con 0x7fa708060870 2021-11-20T05:33:41.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.317+0000 7fa70c863700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa708107d70 con 0x7fa7080f9940 2021-11-20T05:33:41.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.317+0000 7fa70659c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa708060870 0x7fa708107190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:41.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.317+0000 7fa70659c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa708060870 0x7fa708107190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40364/0 (socket says 172.21.15.58:40364) 2021-11-20T05:33:41.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.317+0000 7fa70659c700 1 -- 172.21.15.58:0/457921807 learned_addr learned my addr 172.21.15.58:0/457921807 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:41.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.317+0000 7fa70659c700 1 -- 172.21.15.58:0/457921807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7080f9940 msgr2=0x7fa708107680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:41.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.317+0000 7fa705d9b700 1 --2- 172.21.15.58:0/457921807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7080f9940 0x7fa708107680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:41.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.317+0000 7fa70659c700 1 --2- 172.21.15.58:0/457921807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7080f9940 0x7fa708107680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:41.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.317+0000 7fa70659c700 1 -- 172.21.15.58:0/457921807 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa700009640 con 0x7fa708060870 2021-11-20T05:33:41.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.317+0000 7fa705d9b700 1 --2- 172.21.15.58:0/457921807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7080f9940 0x7fa708107680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:33:41.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.317+0000 7fa70659c700 1 --2- 172.21.15.58:0/457921807 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa708060870 0x7fa708107190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fa70000b620 tx=0x7fa700004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:41.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.317+0000 7fa6feffd700 1 -- 172.21.15.58:0/457921807 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa70000e070 con 0x7fa708060870 2021-11-20T05:33:41.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.318+0000 7fa70c863700 1 -- 172.21.15.58:0/457921807 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7081086b0 con 0x7fa708060870 2021-11-20T05:33:41.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.318+0000 7fa6feffd700 1 -- 172.21.15.58:0/457921807 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa70001d8d0 con 0x7fa708060870 2021-11-20T05:33:41.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.318+0000 7fa6feffd700 1 -- 172.21.15.58:0/457921807 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa700018070 con 0x7fa708060870 2021-11-20T05:33:41.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.318+0000 7fa70c863700 1 -- 172.21.15.58:0/457921807 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa708108b90 con 0x7fa708060870 2021-11-20T05:33:41.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.319+0000 7fa6feffd700 1 -- 172.21.15.58:0/457921807 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa700024060 con 0x7fa708060870 2021-11-20T05:33:41.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.319+0000 7fa6feffd700 1 --2- 172.21.15.58:0/457921807 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa6f405c2a0 0x7fa6f405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:41.326 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.319+0000 7fa6feffd700 1 -- 172.21.15.58:0/457921807 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fa700078520 con 0x7fa708060870 2021-11-20T05:33:41.326 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.319+0000 7fa70c863700 1 -- 172.21.15.58:0/457921807 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ec004fa0 con 0x7fa708060870 2021-11-20T05:33:41.326 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.319+0000 7fa705d9b700 1 --2- 172.21.15.58:0/457921807 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa6f405c2a0 0x7fa6f405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:41.326 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.320+0000 7fa705d9b700 1 --2- 172.21.15.58:0/457921807 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa6f405c2a0 0x7fa6f405e760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fa708060640 tx=0x7fa6f0009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:41.326 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.324+0000 7fa6feffd700 1 -- 172.21.15.58:0/457921807 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa70004b520 con 0x7fa708060870 2021-11-20T05:33:41.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.483+0000 7fa70c863700 1 -- 172.21.15.58:0/457921807 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fa6ec005670 con 0x7fa708060870 2021-11-20T05:33:41.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.483+0000 7fa6feffd700 1 -- 172.21.15.58:0/457921807 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa70004eb40 con 0x7fa708060870 2021-11-20T05:33:41.484 INFO:teuthology.orchestra.run.smithi058.stdout:111669149712 2021-11-20T05:33:41.485 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.485+0000 7fa70c863700 1 -- 172.21.15.58:0/457921807 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa6f405c2a0 msgr2=0x7fa6f405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:41.485 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.485+0000 7fa70c863700 1 --2- 172.21.15.58:0/457921807 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa6f405c2a0 0x7fa6f405e760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fa708060640 tx=0x7fa6f0009040).stop 2021-11-20T05:33:41.485 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.485+0000 7fa70c863700 1 -- 172.21.15.58:0/457921807 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa708060870 msgr2=0x7fa708107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:41.485 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.485+0000 7fa70c863700 1 --2- 172.21.15.58:0/457921807 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa708060870 0x7fa708107190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fa70000b620 tx=0x7fa700004d10).stop 2021-11-20T05:33:41.486 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.485+0000 7fa70c863700 1 -- 172.21.15.58:0/457921807 shutdown_connections 2021-11-20T05:33:41.486 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.485+0000 7fa70c863700 1 --2- 172.21.15.58:0/457921807 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa6f405c2a0 0x7fa6f405e760 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:41.486 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.485+0000 7fa70c863700 1 --2- 172.21.15.58:0/457921807 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa708060870 0x7fa708107190 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:41.486 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.485+0000 7fa70c863700 1 --2- 172.21.15.58:0/457921807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7080f9940 0x7fa708107680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:41.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.485+0000 7fa70c863700 1 -- 172.21.15.58:0/457921807 >> 172.21.15.58:0/457921807 conn(0x7fa7080f0e20 msgr2=0x7fa7080f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:41.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.485+0000 7fa70c863700 1 -- 172.21.15.58:0/457921807 shutdown_connections 2021-11-20T05:33:41.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:41.485+0000 7fa70c863700 1 -- 172.21.15.58:0/457921807 wait complete. 2021-11-20T05:33:41.506 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:41 smithi058 conmon[33537]: cluster 2021-11-20T05:33:40.204368+0000 2021-11-20T05:33:41.506 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:41 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 166 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:41.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:41 smithi187 conmon[36358]: cluster 2021-11-20T05:33:40.204368+0000 mgr.smithi058.zagdlk (mgr.14164) 166 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:41.835 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149709 got 111669149712 for osd.3 2021-11-20T05:33:41.836 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.4 2021-11-20T05:33:42.281 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:42.505 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:42 smithi058 conmon[33537]: audit 2021-11-20T05:33:41.484604+0000 mon.smithi058 (mon.0) 439 : audit [DBG] from='client.? 172.21.15.58:0/457921807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-11-20T05:33:42.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:42 smithi187 conmon[36358]: audit 2021-11-20T05:33:41.484604+0000 mon.smithi058 (mon.0) 439 : audit [DBG] from='client.? 172.21.15.58:0/457921807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-11-20T05:33:43.432 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.432+0000 7f0d05568700 1 -- 172.21.15.58:0/1803815471 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d000f62f0 msgr2=0x7f0d000f6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:43.433 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.432+0000 7f0d05568700 1 --2- 172.21.15.58:0/1803815471 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d000f62f0 0x7f0d000f6750 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f0cf4004660 tx=0x7f0cf4009b30).stop 2021-11-20T05:33:43.434 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.432+0000 7f0d05568700 1 -- 172.21.15.58:0/1803815471 shutdown_connections 2021-11-20T05:33:43.434 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.432+0000 7f0d05568700 1 --2- 172.21.15.58:0/1803815471 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d000f62f0 0x7f0d000f6750 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:43.434 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.432+0000 7f0d05568700 1 --2- 172.21.15.58:0/1803815471 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d000f7ce0 0x7f0d000f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:43.435 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.432+0000 7f0d05568700 1 -- 172.21.15.58:0/1803815471 >> 172.21.15.58:0/1803815471 conn(0x7f0d000f1d70 msgr2=0x7f0d000f41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:43.435 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.432+0000 7f0d05568700 1 -- 172.21.15.58:0/1803815471 shutdown_connections 2021-11-20T05:33:43.435 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.432+0000 7f0d05568700 1 -- 172.21.15.58:0/1803815471 wait complete. 2021-11-20T05:33:43.436 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:43 smithi058 conmon[33537]: cluster 2021-11-20T05:33:42.204773+0000 2021-11-20T05:33:43.436 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:43 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 167 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:43.440 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.433+0000 7f0d05568700 1 Processor -- start 2021-11-20T05:33:43.440 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.433+0000 7f0d05568700 1 -- start start 2021-11-20T05:33:43.441 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.433+0000 7f0d05568700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d000f62f0 0x7f0d0006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:43.441 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.433+0000 7f0d05568700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d000f7ce0 0x7f0d00069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:43.441 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.433+0000 7f0d05568700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0006a380 con 0x7f0d000f62f0 2021-11-20T05:33:43.441 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.433+0000 7f0d05568700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0006a4c0 con 0x7f0d000f7ce0 2021-11-20T05:33:43.443 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.433+0000 7f0cfffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d000f62f0 0x7f0d0006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:43.443 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.433+0000 7f0cfffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d000f62f0 0x7f0d0006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40374/0 (socket says 172.21.15.58:40374) 2021-11-20T05:33:43.443 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.433+0000 7f0cfffff700 1 -- 172.21.15.58:0/1265485050 learned_addr learned my addr 172.21.15.58:0/1265485050 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:43.444 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.434+0000 7f0cff7fe700 1 --2- 172.21.15.58:0/1265485050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d000f7ce0 0x7f0d00069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:43.444 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.434+0000 7f0cfffff700 1 -- 172.21.15.58:0/1265485050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d000f7ce0 msgr2=0x7f0d00069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:43.444 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.434+0000 7f0cfffff700 1 --2- 172.21.15.58:0/1265485050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d000f7ce0 0x7f0d00069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:43.444 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.434+0000 7f0cfffff700 1 -- 172.21.15.58:0/1265485050 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf4005040 con 0x7f0d000f62f0 2021-11-20T05:33:43.445 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.434+0000 7f0cfffff700 1 --2- 172.21.15.58:0/1265485050 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d000f62f0 0x7f0d0006b9a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f0cf000c790 tx=0x7f0cf0007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:43.446 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.434+0000 7f0cfcff9700 1 -- 172.21.15.58:0/1265485050 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cf0013630 con 0x7f0d000f62f0 2021-11-20T05:33:43.446 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.434+0000 7f0cfcff9700 1 -- 172.21.15.58:0/1265485050 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cf0012860 con 0x7f0d000f62f0 2021-11-20T05:33:43.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.434+0000 7f0d05568700 1 -- 172.21.15.58:0/1265485050 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0006a7a0 con 0x7f0d000f62f0 2021-11-20T05:33:43.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.434+0000 7f0cfcff9700 1 -- 172.21.15.58:0/1265485050 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cf00027e0 con 0x7f0d000f62f0 2021-11-20T05:33:43.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.434+0000 7f0d05568700 1 -- 172.21.15.58:0/1265485050 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0010efd0 con 0x7f0d000f62f0 2021-11-20T05:33:43.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.435+0000 7f0d05568700 1 -- 172.21.15.58:0/1265485050 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d00045ad0 con 0x7f0d000f62f0 2021-11-20T05:33:43.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.436+0000 7f0cfcff9700 1 -- 172.21.15.58:0/1265485050 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0cf0010070 con 0x7f0d000f62f0 2021-11-20T05:33:43.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.436+0000 7f0cfcff9700 1 --2- 172.21.15.58:0/1265485050 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0ce805c2a0 0x7f0ce805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:43.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.436+0000 7f0cfcff9700 1 -- 172.21.15.58:0/1265485050 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f0cf0075c40 con 0x7f0d000f62f0 2021-11-20T05:33:43.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.436+0000 7f0cff7fe700 1 --2- 172.21.15.58:0/1265485050 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0ce805c2a0 0x7f0ce805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:43.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.437+0000 7f0cff7fe700 1 --2- 172.21.15.58:0/1265485050 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0ce805c2a0 0x7f0ce805e760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f0cf4004e30 tx=0x7f0cf4009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:43.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.440+0000 7f0cfcff9700 1 -- 172.21.15.58:0/1265485050 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0cf0048be0 con 0x7f0d000f62f0 2021-11-20T05:33:43.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:43 smithi187 conmon[36358]: cluster 2021-11-20T05:33:42.204773+0000 mgr.smithi058.zagdlk (mgr.14164) 167 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:43.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.597+0000 7f0d05568700 1 -- 172.21.15.58:0/1265485050 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f0d00027a00 con 0x7f0d000f62f0 2021-11-20T05:33:43.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.597+0000 7f0cfcff9700 1 -- 172.21.15.58:0/1265485050 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0cf004c200 con 0x7f0d000f62f0 2021-11-20T05:33:43.597 INFO:teuthology.orchestra.run.smithi058.stdout:133143986189 2021-11-20T05:33:43.599 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.599+0000 7f0d05568700 1 -- 172.21.15.58:0/1265485050 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0ce805c2a0 msgr2=0x7f0ce805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:43.599 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.599+0000 7f0d05568700 1 --2- 172.21.15.58:0/1265485050 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0ce805c2a0 0x7f0ce805e760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f0cf4004e30 tx=0x7f0cf4009b30).stop 2021-11-20T05:33:43.599 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.599+0000 7f0d05568700 1 -- 172.21.15.58:0/1265485050 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d000f62f0 msgr2=0x7f0d0006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:43.599 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.599+0000 7f0d05568700 1 --2- 172.21.15.58:0/1265485050 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d000f62f0 0x7f0d0006b9a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f0cf000c790 tx=0x7f0cf0007d90).stop 2021-11-20T05:33:43.600 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.599+0000 7f0d05568700 1 -- 172.21.15.58:0/1265485050 shutdown_connections 2021-11-20T05:33:43.600 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.599+0000 7f0d05568700 1 --2- 172.21.15.58:0/1265485050 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0ce805c2a0 0x7f0ce805e760 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:43.600 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.599+0000 7f0d05568700 1 --2- 172.21.15.58:0/1265485050 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d000f62f0 0x7f0d0006b9a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:43.600 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.599+0000 7f0d05568700 1 --2- 172.21.15.58:0/1265485050 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d000f7ce0 0x7f0d00069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:43.601 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.599+0000 7f0d05568700 1 -- 172.21.15.58:0/1265485050 >> 172.21.15.58:0/1265485050 conn(0x7f0d000f1d70 msgr2=0x7f0d000fb320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:43.601 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.599+0000 7f0d05568700 1 -- 172.21.15.58:0/1265485050 shutdown_connections 2021-11-20T05:33:43.601 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:43.599+0000 7f0d05568700 1 -- 172.21.15.58:0/1265485050 wait complete. 2021-11-20T05:33:43.984 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986187 got 133143986189 for osd.4 2021-11-20T05:33:43.985 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.5 2021-11-20T05:33:44.430 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:44.544 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:44 smithi058 conmon[33537]: audit 2021-11-20T05:33:43.598173 2021-11-20T05:33:44.544 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:44 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 440 : audit [DBG] from='client.? 172.21.15.58:0/1265485050' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-11-20T05:33:44.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:44 smithi187 conmon[36358]: audit 2021-11-20T05:33:43.598173+0000 mon.smithi058 (mon.0) 440 : audit [DBG] from='client.? 172.21.15.58:0/1265485050' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-11-20T05:33:45.479 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.478+0000 7fdda3339700 1 -- 172.21.15.58:0/3920549683 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdd9c0f6980 msgr2=0x7fdd9c0f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:45.480 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.478+0000 7fdda3339700 1 --2- 172.21.15.58:0/3920549683 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdd9c0f6980 0x7fdd9c0f6de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fdd8c004660 tx=0x7fdd8c009b30).stop 2021-11-20T05:33:45.480 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.478+0000 7fdda3339700 1 -- 172.21.15.58:0/3920549683 shutdown_connections 2021-11-20T05:33:45.480 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.478+0000 7fdda3339700 1 --2- 172.21.15.58:0/3920549683 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdd9c0f6980 0x7fdd9c0f6de0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:45.480 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.478+0000 7fdda3339700 1 --2- 172.21.15.58:0/3920549683 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd9c0f56e0 0x7fdd9c0f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:45.481 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.478+0000 7fdda3339700 1 -- 172.21.15.58:0/3920549683 >> 172.21.15.58:0/3920549683 conn(0x7fdd9c0f0e20 msgr2=0x7fdd9c0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:45.481 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.478+0000 7fdda3339700 1 -- 172.21.15.58:0/3920549683 shutdown_connections 2021-11-20T05:33:45.481 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.478+0000 7fdda3339700 1 -- 172.21.15.58:0/3920549683 wait complete. 2021-11-20T05:33:45.481 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.478+0000 7fdda3339700 1 Processor -- start 2021-11-20T05:33:45.482 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.478+0000 7fdda3339700 1 -- start start 2021-11-20T05:33:45.482 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.479+0000 7fdda3339700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd9c0f56e0 0x7fdd9c109dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:45.482 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.479+0000 7fdda3339700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdd9c0f6980 0x7fdd9c10c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:45.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.479+0000 7fdda3339700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c060920 con 0x7fdd9c0f6980 2021-11-20T05:33:45.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.479+0000 7fdda3339700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c060a60 con 0x7fdd9c0f56e0 2021-11-20T05:33:45.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.479+0000 7fdda08d4700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdd9c0f6980 0x7fdd9c10c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:45.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.479+0000 7fdda08d4700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdd9c0f6980 0x7fdd9c10c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40386/0 (socket says 172.21.15.58:40386) 2021-11-20T05:33:45.484 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.479+0000 7fdda08d4700 1 -- 172.21.15.58:0/2872445178 learned_addr learned my addr 172.21.15.58:0/2872445178 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:45.484 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.479+0000 7fdda08d4700 1 -- 172.21.15.58:0/2872445178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd9c0f56e0 msgr2=0x7fdd9c109dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:45.484 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.479+0000 7fdda10d5700 1 --2- 172.21.15.58:0/2872445178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd9c0f56e0 0x7fdd9c109dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:45.484 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.479+0000 7fdda08d4700 1 --2- 172.21.15.58:0/2872445178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd9c0f56e0 0x7fdd9c109dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:45.485 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.479+0000 7fdda08d4700 1 -- 172.21.15.58:0/2872445178 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd8c005040 con 0x7fdd9c0f6980 2021-11-20T05:33:45.485 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.479+0000 7fdda08d4700 1 --2- 172.21.15.58:0/2872445178 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdd9c0f6980 0x7fdd9c10c2c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fdd8c009f50 tx=0x7fdd8c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:45.485 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.479+0000 7fdd91ffb700 1 -- 172.21.15.58:0/2872445178 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd8c018070 con 0x7fdd9c0f6980 2021-11-20T05:33:45.485 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.479+0000 7fdda3339700 1 -- 172.21.15.58:0/2872445178 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9c10c810 con 0x7fdd9c0f6980 2021-11-20T05:33:45.486 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.479+0000 7fdd91ffb700 1 -- 172.21.15.58:0/2872445178 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd8c013410 con 0x7fdd9c0f6980 2021-11-20T05:33:45.486 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.479+0000 7fdd91ffb700 1 -- 172.21.15.58:0/2872445178 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd8c01c3c0 con 0x7fdd9c0f6980 2021-11-20T05:33:45.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.479+0000 7fdda3339700 1 -- 172.21.15.58:0/2872445178 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9c10cd40 con 0x7fdd9c0f6980 2021-11-20T05:33:45.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.481+0000 7fdda3339700 1 -- 172.21.15.58:0/2872445178 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9c045ad0 con 0x7fdd9c0f6980 2021-11-20T05:33:45.487 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:45 smithi058 conmon[33537]: cluster 2021-11-20T05:33:44.205103+0000 2021-11-20T05:33:45.488 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:45 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 168 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:45.489 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.482+0000 7fdd91ffb700 1 -- 172.21.15.58:0/2872445178 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fdd8c01d420 con 0x7fdd9c0f6980 2021-11-20T05:33:45.489 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.482+0000 7fdd91ffb700 1 --2- 172.21.15.58:0/2872445178 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdd8805c2a0 0x7fdd8805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:45.489 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.482+0000 7fdd91ffb700 1 -- 172.21.15.58:0/2872445178 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fdd8c077e80 con 0x7fdd9c0f6980 2021-11-20T05:33:45.489 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.482+0000 7fdda10d5700 1 --2- 172.21.15.58:0/2872445178 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdd8805c2a0 0x7fdd8805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:45.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.483+0000 7fdda10d5700 1 --2- 172.21.15.58:0/2872445178 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdd8805c2a0 0x7fdd8805e760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fdd9c0605a0 tx=0x7fdd9800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:45.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.486+0000 7fdd91ffb700 1 -- 172.21.15.58:0/2872445178 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd8c04ee60 con 0x7fdd9c0f6980 2021-11-20T05:33:45.644 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.644+0000 7fdda3339700 1 -- 172.21.15.58:0/2872445178 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fdd9c027a00 con 0x7fdd9c0f6980 2021-11-20T05:33:45.644 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.644+0000 7fdd91ffb700 1 -- 172.21.15.58:0/2872445178 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fdd8c04af70 con 0x7fdd9c0f6980 2021-11-20T05:33:45.645 INFO:teuthology.orchestra.run.smithi058.stdout:154618822666 2021-11-20T05:33:45.646 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.646+0000 7fdda3339700 1 -- 172.21.15.58:0/2872445178 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdd8805c2a0 msgr2=0x7fdd8805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:45.646 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.646+0000 7fdda3339700 1 --2- 172.21.15.58:0/2872445178 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdd8805c2a0 0x7fdd8805e760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fdd9c0605a0 tx=0x7fdd9800b040).stop 2021-11-20T05:33:45.646 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.646+0000 7fdda3339700 1 -- 172.21.15.58:0/2872445178 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdd9c0f6980 msgr2=0x7fdd9c10c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:45.647 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.646+0000 7fdda3339700 1 --2- 172.21.15.58:0/2872445178 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdd9c0f6980 0x7fdd9c10c2c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fdd8c009f50 tx=0x7fdd8c009b30).stop 2021-11-20T05:33:45.647 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.646+0000 7fdda3339700 1 -- 172.21.15.58:0/2872445178 shutdown_connections 2021-11-20T05:33:45.647 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.646+0000 7fdda3339700 1 --2- 172.21.15.58:0/2872445178 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdd8805c2a0 0x7fdd8805e760 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:45.647 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.646+0000 7fdda3339700 1 --2- 172.21.15.58:0/2872445178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd9c0f56e0 0x7fdd9c109dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:45.648 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.646+0000 7fdda3339700 1 --2- 172.21.15.58:0/2872445178 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdd9c0f6980 0x7fdd9c10c2c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:45.648 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.646+0000 7fdda3339700 1 -- 172.21.15.58:0/2872445178 >> 172.21.15.58:0/2872445178 conn(0x7fdd9c0f0e20 msgr2=0x7fdd9c0fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:45.648 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.646+0000 7fdda3339700 1 -- 172.21.15.58:0/2872445178 shutdown_connections 2021-11-20T05:33:45.648 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:45.646+0000 7fdda3339700 1 -- 172.21.15.58:0/2872445178 wait complete. 2021-11-20T05:33:45.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:45 smithi187 conmon[36358]: cluster 2021-11-20T05:33:44.205103+0000 mgr.smithi058.zagdlk (mgr.14164) 168 : 2021-11-20T05:33:45.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:45 smithi187 conmon[36358]: cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:46.028 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822664 got 154618822666 for osd.5 2021-11-20T05:33:46.029 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2021-11-20T05:33:46.029 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph pg dump --format=json 2021-11-20T05:33:46.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:46 smithi058 conmon[33537]: audit 2021-11-20T05:33:45.645455+0000 2021-11-20T05:33:46.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:46 smithi058 conmon[33537]: mon.smithi058 (mon.0) 441 : audit [DBG] from='client.? 172.21.15.58:0/2872445178' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-11-20T05:33:46.476 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:46.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:46 smithi187 conmon[36358]: audit 2021-11-20T05:33:45.645455+0000 mon.smithi058 (mon.0) 441 : audit [DBG] from='client.? 172.21.15.58:0/2872445178' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-11-20T05:33:47.335 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:47 smithi058 conmon[33537]: cluster 2021-11-20T05:33:46.205459+0000 mgr.smithi058.zagdlk (mgr.14164) 169 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:47.662 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.661+0000 7f7e688f1700 1 -- 172.21.15.58:0/1124921077 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7e640f9780 msgr2=0x7f7e640fbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:47.662 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.661+0000 7f7e688f1700 1 --2- 172.21.15.58:0/1124921077 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7e640f9780 0x7f7e640fbbd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f7e54004660 tx=0x7f7e54009b30).stop 2021-11-20T05:33:47.662 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.661+0000 7f7e688f1700 1 -- 172.21.15.58:0/1124921077 shutdown_connections 2021-11-20T05:33:47.663 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.661+0000 7f7e688f1700 1 --2- 172.21.15.58:0/1124921077 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7e640f9780 0x7f7e640fbbd0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:47.663 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.661+0000 7f7e688f1700 1 --2- 172.21.15.58:0/1124921077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e64060870 0x7f7e640f9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:33:47.663 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.661+0000 7f7e688f1700 1 -- 172.21.15.58:0/1124921077 >> 172.21.15.58:0/1124921077 conn(0x7f7e640f0ca0 msgr2=0x7f7e640f30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:47.664 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.662+0000 7f7e688f1700 1 -- 172.21.15.58:0/1124921077 shutdown_connections 2021-11-20T05:33:47.664 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.662+0000 7f7e688f1700 1 -- 172.21.15.58:0/1124921077 wait complete. 2021-11-20T05:33:47.664 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.662+0000 7f7e688f1700 1 Processor -- start 2021-11-20T05:33:47.664 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.662+0000 7f7e688f1700 1 -- start start 2021-11-20T05:33:47.667 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.663+0000 7f7e688f1700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7e64060870 0x7f7e64106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:47.667 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.663+0000 7f7e688f1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e640f9780 0x7f7e641073e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:47.667 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.663+0000 7f7e688f1700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e64107a20 con 0x7f7e64060870 2021-11-20T05:33:47.668 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.663+0000 7f7e688f1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e641082d0 con 0x7f7e640f9780 2021-11-20T05:33:47.668 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.663+0000 7f7e637fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7e64060870 0x7f7e64106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:47.668 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.663+0000 7f7e637fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7e64060870 0x7f7e64106ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40394/0 (socket says 172.21.15.58:40394) 2021-11-20T05:33:47.668 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.663+0000 7f7e637fe700 1 -- 172.21.15.58:0/2785150685 learned_addr learned my addr 172.21.15.58:0/2785150685 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:47.668 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.663+0000 7f7e62ffd700 1 --2- 172.21.15.58:0/2785150685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e640f9780 0x7f7e641073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:47.669 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.663+0000 7f7e637fe700 1 -- 172.21.15.58:0/2785150685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e640f9780 msgr2=0x7f7e641073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:47.669 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.663+0000 7f7e637fe700 1 --2- 172.21.15.58:0/2785150685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e640f9780 0x7f7e641073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:47.669 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.663+0000 7f7e637fe700 1 -- 172.21.15.58:0/2785150685 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e54005040 con 0x7f7e64060870 2021-11-20T05:33:47.669 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.663+0000 7f7e637fe700 1 --2- 172.21.15.58:0/2785150685 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7e64060870 0x7f7e64106ef0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f7e5800b410 tx=0x7f7e58006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:47.670 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.663+0000 7f7e4bfff700 1 -- 172.21.15.58:0/2785150685 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e580125f0 con 0x7f7e64060870 2021-11-20T05:33:47.670 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.663+0000 7f7e688f1700 1 -- 172.21.15.58:0/2785150685 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e641085b0 con 0x7f7e64060870 2021-11-20T05:33:47.671 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.663+0000 7f7e4bfff700 1 -- 172.21.15.58:0/2785150685 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e58012c00 con 0x7f7e64060870 2021-11-20T05:33:47.671 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.663+0000 7f7e4bfff700 1 -- 172.21.15.58:0/2785150685 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e58002aa0 con 0x7f7e64060870 2021-11-20T05:33:47.671 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.664+0000 7f7e688f1700 1 -- 172.21.15.58:0/2785150685 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e64108b70 con 0x7f7e64060870 2021-11-20T05:33:47.671 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.665+0000 7f7e4bfff700 1 -- 172.21.15.58:0/2785150685 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7e58012d70 con 0x7f7e64060870 2021-11-20T05:33:47.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.665+0000 7f7e688f1700 1 -- 172.21.15.58:0/2785150685 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e64059100 con 0x7f7e64060870 2021-11-20T05:33:47.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.665+0000 7f7e4bfff700 1 --2- 172.21.15.58:0/2785150685 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7e4c05c2b0 0x7f7e4c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:47.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.665+0000 7f7e4bfff700 1 -- 172.21.15.58:0/2785150685 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7e58075d70 con 0x7f7e64060870 2021-11-20T05:33:47.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.665+0000 7f7e62ffd700 1 --2- 172.21.15.58:0/2785150685 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7e4c05c2b0 0x7f7e4c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:47.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.666+0000 7f7e62ffd700 1 --2- 172.21.15.58:0/2785150685 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7e4c05c2b0 0x7f7e4c05e770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f7e54004e30 tx=0x7f7e54009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:47.674 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.669+0000 7f7e4bfff700 1 -- 172.21.15.58:0/2785150685 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e58048d10 con 0x7f7e64060870 2021-11-20T05:33:47.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:47 smithi187 conmon[36358]: cluster 2021-11-20T05:33:46.205459+0000 mgr.smithi058.zagdlk (mgr.14164) 169 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:47.822 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.822+0000 7f7e688f1700 1 -- 172.21.15.58:0/2785150685 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7e640feef0 con 0x7f7e4c05c2b0 2021-11-20T05:33:47.823 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.824+0000 7f7e4bfff700 1 -- 172.21.15.58:0/2785150685 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+24179 (secure 0 0 0) 0x7f7e640feef0 con 0x7f7e4c05c2b0 2021-11-20T05:33:47.824 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:33:47.825 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.825+0000 7f7e688f1700 1 -- 172.21.15.58:0/2785150685 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7e4c05c2b0 msgr2=0x7f7e4c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:47.825 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.825+0000 7f7e688f1700 1 --2- 172.21.15.58:0/2785150685 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7e4c05c2b0 0x7f7e4c05e770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f7e54004e30 tx=0x7f7e54009b30).stop 2021-11-20T05:33:47.825 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.825+0000 7f7e688f1700 1 -- 172.21.15.58:0/2785150685 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7e64060870 msgr2=0x7f7e64106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:47.826 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.825+0000 7f7e688f1700 1 --2- 172.21.15.58:0/2785150685 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7e64060870 0x7f7e64106ef0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f7e5800b410 tx=0x7f7e58006ef0).stop 2021-11-20T05:33:47.826 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.825+0000 7f7e688f1700 1 -- 172.21.15.58:0/2785150685 shutdown_connections 2021-11-20T05:33:47.826 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.825+0000 7f7e688f1700 1 --2- 172.21.15.58:0/2785150685 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7e4c05c2b0 0x7f7e4c05e770 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:47.826 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.825+0000 7f7e688f1700 1 --2- 172.21.15.58:0/2785150685 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7e64060870 0x7f7e64106ef0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:47.826 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.825+0000 7f7e688f1700 1 --2- 172.21.15.58:0/2785150685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e640f9780 0x7f7e641073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:47.827 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.825+0000 7f7e688f1700 1 -- 172.21.15.58:0/2785150685 >> 172.21.15.58:0/2785150685 conn(0x7f7e640f0ca0 msgr2=0x7f7e640f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:47.827 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.825+0000 7f7e688f1700 1 -- 172.21.15.58:0/2785150685 shutdown_connections 2021-11-20T05:33:47.828 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:47.825+0000 7f7e688f1700 1 -- 172.21.15.58:0/2785150685 wait complete. 2021-11-20T05:33:47.828 INFO:teuthology.orchestra.run.smithi058.stderr:dumped all 2021-11-20T05:33:48.181 INFO:teuthology.orchestra.run.smithi058.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2021-11-20T05:33:46.205265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":31112,"kb_used_data":1944,"kb_used_omap":0,"kb_used_meta":29120,"kb_avail":562439800,"statfs":{"total":575970213888,"available":575938355200,"internally_reserved":0,"allocated":1990656,"data_stored":745914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29818880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"35","reported_epoch":"36","state":"active+clean","last_fresh":"2021-11-20T05:33:10.129157+0000","last_change":"2021-11-20T05:32:52.007542+0000","last_active":"2021-11-20T05:33:10.129157+0000","last_peered":"2021-11-20T05:33:10.129157+0000","last_clean":"2021-11-20T05:33:10.129157+0000","last_became_active":"2021-11-20T05:32:52.007253+0000","last_became_peered":"2021-11-20T05:32:52.007253+0000","last_unstale":"2021-11-20T05:33:10.129157+0000","last_undegraded":"2021-11-20T05:33:10.129157+0000","last_fullsized":"2021-11-20T05:33:10.129157+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-11-20T05:32:27.605385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-11-20T05:32:27.605385+0000","last_clean_scrub_stamp":"2021-11-20T05:32:27.605385+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":36,"seq":154618822667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5068,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740084,"statfs":{"total":95995035648,"available":95989846016,"internally_reserved":0,"allocated":331776,"data_stored":124319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5132,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740020,"statfs":{"total":95995035648,"available":95989780480,"internally_reserved":0,"allocated":331776,"data_stored":124319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5196,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739956,"statfs":{"total":95995035648,"available":95989714944,"internally_reserved":0,"allocated":331776,"data_stored":124319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 05:33:34 2021","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.60799999999999998}]},{"osd":1,"last update":"Sat Nov 20 05:33:34 2021","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.61699999999999999}]},{"osd":2,"last update":"Sat Nov 20 05:33:34 2021","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]}]},{"osd":2,"up_from":19,"seq":81604378643,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5196,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739956,"statfs":{"total":95995035648,"available":95989714944,"internally_reserved":0,"allocated":331776,"data_stored":124319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 05:33:17 2021","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"last":0.30099999999999999}]},{"osd":1,"last update":"Sat Nov 20 05:33:17 2021","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.37}]},{"osd":3,"last update":"Sat Nov 20 05:33:38 2021","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.68500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705691,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5260,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739892,"statfs":{"total":95995035648,"available":95989649408,"internally_reserved":0,"allocated":331776,"data_stored":124319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Nov 20 05:33:02 2021","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.34100000000000003}]},{"osd":2,"last update":"Sat Nov 20 05:33:25 2021","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"last":0.307}]},{"osd":3,"last update":"Sat Nov 20 05:33:33 2021","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.68899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542169,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5260,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739892,"statfs":{"total":95995035648,"available":95989649408,"internally_reserved":0,"allocated":331776,"data_stored":124319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 05:32:59 2021","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.34599999999999997}]},{"osd":2,"last update":"Sat Nov 20 05:33:19 2021","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.38200000000000001}]},{"osd":3,"last update":"Sat Nov 20 05:33:37 2021","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":0.58999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-11-20T05:33:48.183 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph pg dump --format=json 2021-11-20T05:33:48.629 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:49.334 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:49 smithi058 conmon[33537]: audit 2021-11-20T05:33:47.823645+0000 mgr.smithi058.zagdlk (mgr.14164) 170 : audit [DBG] from='client.14388 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T05:33:49.334 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:49 smithi058 conmon[33537]: cluster 2021-11-20T05:33:48.205780+0000 mgr.smithi058.zagdlk (mgr.14164) 171 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:49.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:49 smithi187 conmon[36358]: audit 2021-11-20T05:33:47.823645+0000 mgr.smithi058.zagdlk (mgr.14164) 170 : audit [DBG] from='client.14388 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T05:33:49.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:49 smithi187 conmon[36358]: cluster 2021-11-20T05:33:48.205780+0000 mgr.smithi058.zagdlk (mgr.14164) 171 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:50.005 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.002+0000 7f93d335c700 1 -- 172.21.15.58:0/3777256761 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f93cc0f6960 msgr2=0x7f93cc0f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:50.005 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.002+0000 7f93d335c700 1 --2- 172.21.15.58:0/3777256761 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f93cc0f6960 0x7f93cc0f6dc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f93c8004660 tx=0x7f93c8009b30).stop 2021-11-20T05:33:50.005 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.002+0000 7f93d335c700 1 -- 172.21.15.58:0/3777256761 shutdown_connections 2021-11-20T05:33:50.006 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.002+0000 7f93d335c700 1 --2- 172.21.15.58:0/3777256761 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f93cc0f6960 0x7f93cc0f6dc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:50.006 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.002+0000 7f93d335c700 1 --2- 172.21.15.58:0/3777256761 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93cc0f56c0 0x7f93cc0f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:50.006 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.002+0000 7f93d335c700 1 -- 172.21.15.58:0/3777256761 >> 172.21.15.58:0/3777256761 conn(0x7f93cc0f0e20 msgr2=0x7f93cc0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:50.006 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.002+0000 7f93d335c700 1 -- 172.21.15.58:0/3777256761 shutdown_connections 2021-11-20T05:33:50.006 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.002+0000 7f93d335c700 1 -- 172.21.15.58:0/3777256761 wait complete. 2021-11-20T05:33:50.007 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.002+0000 7f93d335c700 1 Processor -- start 2021-11-20T05:33:50.007 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.003+0000 7f93d335c700 1 -- start start 2021-11-20T05:33:50.007 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.003+0000 7f93d335c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93cc0f56c0 0x7f93cc10c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:50.007 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.003+0000 7f93d335c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f93cc0f6960 0x7f93cc10e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:50.008 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.003+0000 7f93d335c700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc060810 con 0x7f93cc0f6960 2021-11-20T05:33:50.008 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.003+0000 7f93d335c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc060950 con 0x7f93cc0f56c0 2021-11-20T05:33:50.008 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.003+0000 7f93d08f7700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f93cc0f6960 0x7f93cc10e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:50.008 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.003+0000 7f93d08f7700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f93cc0f6960 0x7f93cc10e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40406/0 (socket says 172.21.15.58:40406) 2021-11-20T05:33:50.008 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.003+0000 7f93d08f7700 1 -- 172.21.15.58:0/1855275644 learned_addr learned my addr 172.21.15.58:0/1855275644 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:50.009 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.003+0000 7f93d08f7700 1 -- 172.21.15.58:0/1855275644 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93cc0f56c0 msgr2=0x7f93cc10c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:50.009 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.003+0000 7f93d10f8700 1 --2- 172.21.15.58:0/1855275644 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93cc0f56c0 0x7f93cc10c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:50.009 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.003+0000 7f93d08f7700 1 --2- 172.21.15.58:0/1855275644 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93cc0f56c0 0x7f93cc10c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:50.009 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.003+0000 7f93d08f7700 1 -- 172.21.15.58:0/1855275644 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c8005040 con 0x7f93cc0f6960 2021-11-20T05:33:50.010 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.003+0000 7f93d10f8700 1 --2- 172.21.15.58:0/1855275644 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93cc0f56c0 0x7f93cc10c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:33:50.010 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.003+0000 7f93d08f7700 1 --2- 172.21.15.58:0/1855275644 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f93cc0f6960 0x7f93cc10e700 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f93c8009830 tx=0x7f93c8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:50.010 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.003+0000 7f93c1ffb700 1 -- 172.21.15.58:0/1855275644 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c8018070 con 0x7f93cc0f6960 2021-11-20T05:33:50.010 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.003+0000 7f93d335c700 1 -- 172.21.15.58:0/1855275644 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93cc10ebf0 con 0x7f93cc0f6960 2021-11-20T05:33:50.010 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.004+0000 7f93d335c700 1 -- 172.21.15.58:0/1855275644 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93cc10efc0 con 0x7f93cc0f6960 2021-11-20T05:33:50.011 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.004+0000 7f93c1ffb700 1 -- 172.21.15.58:0/1855275644 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93c8013410 con 0x7f93cc0f6960 2021-11-20T05:33:50.011 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.004+0000 7f93c1ffb700 1 -- 172.21.15.58:0/1855275644 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c801c3e0 con 0x7f93cc0f6960 2021-11-20T05:33:50.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.005+0000 7f93d335c700 1 -- 172.21.15.58:0/1855275644 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ac004fa0 con 0x7f93cc0f6960 2021-11-20T05:33:50.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.005+0000 7f93c1ffb700 1 -- 172.21.15.58:0/1855275644 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f93c801d420 con 0x7f93cc0f6960 2021-11-20T05:33:50.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.006+0000 7f93c1ffb700 1 --2- 172.21.15.58:0/1855275644 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f93b805c2a0 0x7f93b805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:50.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.006+0000 7f93c1ffb700 1 -- 172.21.15.58:0/1855275644 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f93c80779d0 con 0x7f93cc0f6960 2021-11-20T05:33:50.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.006+0000 7f93d10f8700 1 --2- 172.21.15.58:0/1855275644 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f93b805c2a0 0x7f93b805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:50.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.006+0000 7f93d10f8700 1 --2- 172.21.15.58:0/1855275644 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f93b805c2a0 0x7f93b805e760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f93bc007fd0 tx=0x7f93bc009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:50.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.010+0000 7f93c1ffb700 1 -- 172.21.15.58:0/1855275644 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93c804a9d0 con 0x7f93cc0f6960 2021-11-20T05:33:50.164 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.163+0000 7f93d335c700 1 -- 172.21.15.58:0/1855275644 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f93ac000bc0 con 0x7f93b805c2a0 2021-11-20T05:33:50.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.164+0000 7f93c1ffb700 1 -- 172.21.15.58:0/1855275644 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+24194 (secure 0 0 0) 0x7f93ac000bc0 con 0x7f93b805c2a0 2021-11-20T05:33:50.166 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:33:50.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.166+0000 7f93d335c700 1 -- 172.21.15.58:0/1855275644 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f93b805c2a0 msgr2=0x7f93b805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:50.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.166+0000 7f93d335c700 1 --2- 172.21.15.58:0/1855275644 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f93b805c2a0 0x7f93b805e760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f93bc007fd0 tx=0x7f93bc009040).stop 2021-11-20T05:33:50.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.166+0000 7f93d335c700 1 -- 172.21.15.58:0/1855275644 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f93cc0f6960 msgr2=0x7f93cc10e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:50.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.166+0000 7f93d335c700 1 --2- 172.21.15.58:0/1855275644 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f93cc0f6960 0x7f93cc10e700 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f93c8009830 tx=0x7f93c8009b30).stop 2021-11-20T05:33:50.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.166+0000 7f93d335c700 1 -- 172.21.15.58:0/1855275644 shutdown_connections 2021-11-20T05:33:50.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.166+0000 7f93d335c700 1 --2- 172.21.15.58:0/1855275644 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f93b805c2a0 0x7f93b805e760 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:50.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.166+0000 7f93d335c700 1 --2- 172.21.15.58:0/1855275644 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93cc0f56c0 0x7f93cc10c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:50.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.166+0000 7f93d335c700 1 --2- 172.21.15.58:0/1855275644 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f93cc0f6960 0x7f93cc10e700 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:50.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.166+0000 7f93d335c700 1 -- 172.21.15.58:0/1855275644 >> 172.21.15.58:0/1855275644 conn(0x7f93cc0f0e20 msgr2=0x7f93cc0fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:50.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.166+0000 7f93d335c700 1 -- 172.21.15.58:0/1855275644 shutdown_connections 2021-11-20T05:33:50.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:50.166+0000 7f93d335c700 1 -- 172.21.15.58:0/1855275644 wait complete. 2021-11-20T05:33:50.170 INFO:teuthology.orchestra.run.smithi058.stderr:dumped all 2021-11-20T05:33:50.533 INFO:teuthology.orchestra.run.smithi058.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2021-11-20T05:33:48.205650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":31112,"kb_used_data":1944,"kb_used_omap":0,"kb_used_meta":29120,"kb_avail":562439800,"statfs":{"total":575970213888,"available":575938355200,"internally_reserved":0,"allocated":1990656,"data_stored":745914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29818880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"35","reported_epoch":"36","state":"active+clean","last_fresh":"2021-11-20T05:33:10.129157+0000","last_change":"2021-11-20T05:32:52.007542+0000","last_active":"2021-11-20T05:33:10.129157+0000","last_peered":"2021-11-20T05:33:10.129157+0000","last_clean":"2021-11-20T05:33:10.129157+0000","last_became_active":"2021-11-20T05:32:52.007253+0000","last_became_peered":"2021-11-20T05:32:52.007253+0000","last_unstale":"2021-11-20T05:33:10.129157+0000","last_undegraded":"2021-11-20T05:33:10.129157+0000","last_fullsized":"2021-11-20T05:33:10.129157+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-11-20T05:32:27.605385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-11-20T05:32:27.605385+0000","last_clean_scrub_stamp":"2021-11-20T05:32:27.605385+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":36,"seq":154618822667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5068,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740084,"statfs":{"total":95995035648,"available":95989846016,"internally_reserved":0,"allocated":331776,"data_stored":124319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986191,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5132,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740020,"statfs":{"total":95995035648,"available":95989780480,"internally_reserved":0,"allocated":331776,"data_stored":124319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5196,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739956,"statfs":{"total":95995035648,"available":95989714944,"internally_reserved":0,"allocated":331776,"data_stored":124319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 05:33:34 2021","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.60799999999999998}]},{"osd":1,"last update":"Sat Nov 20 05:33:34 2021","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.61699999999999999}]},{"osd":2,"last update":"Sat Nov 20 05:33:34 2021","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]}]},{"osd":2,"up_from":19,"seq":81604378644,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5196,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739956,"statfs":{"total":95995035648,"available":95989714944,"internally_reserved":0,"allocated":331776,"data_stored":124319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 05:33:17 2021","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"last":0.28599999999999998}]},{"osd":1,"last update":"Sat Nov 20 05:33:17 2021","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.31900000000000001}]},{"osd":3,"last update":"Sat Nov 20 05:33:38 2021","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705692,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5260,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739892,"statfs":{"total":95995035648,"available":95989649408,"internally_reserved":0,"allocated":331776,"data_stored":124319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Nov 20 05:33:02 2021","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.217}]},{"osd":2,"last update":"Sat Nov 20 05:33:25 2021","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"last":0.26200000000000001}]},{"osd":3,"last update":"Sat Nov 20 05:33:33 2021","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.68300000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542169,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5260,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739892,"statfs":{"total":95995035648,"available":95989649408,"internally_reserved":0,"allocated":331776,"data_stored":124319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 05:32:59 2021","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.34599999999999997}]},{"osd":2,"last update":"Sat Nov 20 05:33:19 2021","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.38200000000000001}]},{"osd":3,"last update":"Sat Nov 20 05:33:37 2021","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":0.58999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-11-20T05:33:50.534 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2021-11-20T05:33:50.535 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-11-20T05:33:50.535 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2021-11-20T05:33:50.535 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- ceph health --format=json 2021-11-20T05:33:50.979 INFO:teuthology.orchestra.run.smithi058.stderr:Inferring config /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/mon.smithi058/config 2021-11-20T05:33:51.442 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:51 smithi058 conmon[33537]: audit 2021-11-20T05:33:50.165576+0000 mgr.smithi058.zagdlk (mgr.14164) 172 : audit [DBG] from='client.14392 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T05:33:51.442 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:51 smithi058 conmon[33537]: cluster 2021-11-20T05:33:50.206072+0000 mgr.smithi058.zagdlk (mgr.14164) 173 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:51.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:51 smithi187 conmon[36358]: audit 2021-11-20T05:33:50.165576+0000 mgr.smithi058.zagdlk (mgr.14164) 172 : audit [DBG] from='client.14392 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T05:33:51.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:51 smithi187 conmon[36358]: cluster 2021-11-20 2021-11-20T05:33:51.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:51 smithi187 conmon[36358]: T05:33:50.206072+0000 mgr.smithi058.zagdlk (mgr.14164) 173 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:52.036 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.034+0000 7f61c67d6700 1 -- 172.21.15.58:0/528065054 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f61c00f67e0 msgr2=0x7f61c00f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:52.036 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.034+0000 7f61c67d6700 1 --2- 172.21.15.58:0/528065054 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f61c00f67e0 0x7f61c00f6c00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f61b0004660 tx=0x7f61b0009b30).stop 2021-11-20T05:33:52.036 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.034+0000 7f61c67d6700 1 -- 172.21.15.58:0/528065054 shutdown_connections 2021-11-20T05:33:52.037 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.034+0000 7f61c67d6700 1 --2- 172.21.15.58:0/528065054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61c00f7a80 0x7f61c00f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:33:52.037 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.034+0000 7f61c67d6700 1 --2- 172.21.15.58:0/528065054 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f61c00f67e0 0x7f61c00f6c00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:52.037 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.034+0000 7f61c67d6700 1 -- 172.21.15.58:0/528065054 >> 172.21.15.58:0/528065054 conn(0x7f61c00f1f20 msgr2=0x7f61c00f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:52.038 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.034+0000 7f61c67d6700 1 -- 172.21.15.58:0/528065054 shutdown_connections 2021-11-20T05:33:52.039 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.034+0000 7f61c67d6700 1 -- 172.21.15.58:0/528065054 wait complete. 2021-11-20T05:33:52.039 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.034+0000 7f61c67d6700 1 Processor -- start 2021-11-20T05:33:52.039 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.034+0000 7f61c67d6700 1 -- start start 2021-11-20T05:33:52.039 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.035+0000 7f61c67d6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61c00f67e0 0x7f61c01070e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:52.040 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.035+0000 7f61c67d6700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f61c00f7a80 0x7f61c01075d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:52.040 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.035+0000 7f61c67d6700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c0107c10 con 0x7f61c00f7a80 2021-11-20T05:33:52.040 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.035+0000 7f61c67d6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c01084c0 con 0x7f61c00f67e0 2021-11-20T05:33:52.040 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.035+0000 7f61bf7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f61c00f7a80 0x7f61c01075d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:52.041 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.035+0000 7f61bf7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f61c00f7a80 0x7f61c01075d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40416/0 (socket says 172.21.15.58:40416) 2021-11-20T05:33:52.041 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.035+0000 7f61bf7fe700 1 -- 172.21.15.58:0/1528051435 learned_addr learned my addr 172.21.15.58:0/1528051435 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:52.041 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.035+0000 7f61bffff700 1 --2- 172.21.15.58:0/1528051435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61c00f67e0 0x7f61c01070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:52.042 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.035+0000 7f61bf7fe700 1 -- 172.21.15.58:0/1528051435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61c00f67e0 msgr2=0x7f61c01070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:52.042 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.035+0000 7f61bf7fe700 1 --2- 172.21.15.58:0/1528051435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61c00f67e0 0x7f61c01070e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:52.042 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.035+0000 7f61bf7fe700 1 -- 172.21.15.58:0/1528051435 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b0005040 con 0x7f61c00f7a80 2021-11-20T05:33:52.042 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.035+0000 7f61bf7fe700 1 --2- 172.21.15.58:0/1528051435 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f61c00f7a80 0x7f61c01075d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f61a800b410 tx=0x7f61a8006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:52.043 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.035+0000 7f61bcff9700 1 -- 172.21.15.58:0/1528051435 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61a80125f0 con 0x7f61c00f7a80 2021-11-20T05:33:52.043 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.035+0000 7f61c67d6700 1 -- 172.21.15.58:0/1528051435 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c01087a0 con 0x7f61c00f7a80 2021-11-20T05:33:52.043 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.035+0000 7f61bcff9700 1 -- 172.21.15.58:0/1528051435 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61a800be30 con 0x7f61c00f7a80 2021-11-20T05:33:52.043 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.035+0000 7f61bcff9700 1 -- 172.21.15.58:0/1528051435 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61a8002a20 con 0x7f61c00f7a80 2021-11-20T05:33:52.044 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.035+0000 7f61c67d6700 1 -- 172.21.15.58:0/1528051435 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c0108d60 con 0x7f61c00f7a80 2021-11-20T05:33:52.045 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.036+0000 7f61c67d6700 1 -- 172.21.15.58:0/1528051435 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c0059100 con 0x7f61c00f7a80 2021-11-20T05:33:52.045 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.037+0000 7f61bcff9700 1 -- 172.21.15.58:0/1528051435 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f61a800f030 con 0x7f61c00f7a80 2021-11-20T05:33:52.045 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.037+0000 7f61bcff9700 1 --2- 172.21.15.58:0/1528051435 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f61ac05c2a0 0x7f61ac05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:52.045 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.037+0000 7f61bcff9700 1 -- 172.21.15.58:0/1528051435 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f61a8075f70 con 0x7f61c00f7a80 2021-11-20T05:33:52.046 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.037+0000 7f61bffff700 1 --2- 172.21.15.58:0/1528051435 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f61ac05c2a0 0x7f61ac05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:52.046 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.037+0000 7f61bffff700 1 --2- 172.21.15.58:0/1528051435 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f61ac05c2a0 0x7f61ac05e760 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f61b0004630 tx=0x7f61b0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:52.046 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.041+0000 7f61bcff9700 1 -- 172.21.15.58:0/1528051435 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61a8048f70 con 0x7f61c00f7a80 2021-11-20T05:33:52.265 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.264+0000 7f61c67d6700 1 -- 172.21.15.58:0/1528051435 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f61c00feff0 con 0x7f61c00f7a80 2021-11-20T05:33:52.265 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.264+0000 7f61bcff9700 1 -- 172.21.15.58:0/1528051435 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7f61a804c590 con 0x7f61c00f7a80 2021-11-20T05:33:52.266 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:33:52.268 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.266+0000 7f61c67d6700 1 -- 172.21.15.58:0/1528051435 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f61ac05c2a0 msgr2=0x7f61ac05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:52.268 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.266+0000 7f61c67d6700 1 --2- 172.21.15.58:0/1528051435 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f61ac05c2a0 0x7f61ac05e760 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f61b0004630 tx=0x7f61b0009b30).stop 2021-11-20T05:33:52.268 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.266+0000 7f61c67d6700 1 -- 172.21.15.58:0/1528051435 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f61c00f7a80 msgr2=0x7f61c01075d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:52.269 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.266+0000 7f61c67d6700 1 --2- 172.21.15.58:0/1528051435 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f61c00f7a80 0x7f61c01075d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f61a800b410 tx=0x7f61a8006ef0).stop 2021-11-20T05:33:52.269 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.266+0000 7f61c67d6700 1 -- 172.21.15.58:0/1528051435 shutdown_connections 2021-11-20T05:33:52.270 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.266+0000 7f61c67d6700 1 --2- 172.21.15.58:0/1528051435 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f61ac05c2a0 0x7f61ac05e760 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:52.270 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.266+0000 7f61c67d6700 1 --2- 172.21.15.58:0/1528051435 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61c00f67e0 0x7f61c01070e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:52.270 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.266+0000 7f61c67d6700 1 --2- 172.21.15.58:0/1528051435 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f61c00f7a80 0x7f61c01075d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:52.271 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.266+0000 7f61c67d6700 1 -- 172.21.15.58:0/1528051435 >> 172.21.15.58:0/1528051435 conn(0x7f61c00f1f20 msgr2=0x7f61c00fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:52.271 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.266+0000 7f61c67d6700 1 -- 172.21.15.58:0/1528051435 shutdown_connections 2021-11-20T05:33:52.271 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:52.266+0000 7f61c67d6700 1 -- 172.21.15.58:0/1528051435 wait complete. 2021-11-20T05:33:52.679 INFO:teuthology.orchestra.run.smithi058.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-11-20T05:33:52.680 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2021-11-20T05:33:52.681 INFO:tasks.cephadm:Setup complete, yielding 2021-11-20T05:33:52.681 INFO:teuthology.run_tasks:Running task print... 2021-11-20T05:33:52.694 INFO:teuthology.task.print:**** done starting v16.2.4 2021-11-20T05:33:52.695 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T05:33:52.773 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi058.front.sepia.ceph.com 2021-11-20T05:33:52.774 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch status' 2021-11-20T05:33:53.579 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:53 smithi187 conmon[36358]: cluster 2021-11-20T05:33:52.206387+0000 mgr.smithi058.zagdlk (mgr.14164) 174 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:53.579 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:53 smithi187 conmon[36358]: audit 2021-11-20T05:33:52.266426+0000 mon.smithi058 (mon.0) 442 : audit [DBG] from='client.? 172.21.15.58:0/1528051435' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-11-20T05:33:53.583 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:53 smithi058 conmon[33537]: cluster 2021-11-20T05:33:52.206387 2021-11-20T05:33:53.583 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:53 smithi058 conmon[33537]: +0000 mgr.smithi058.zagdlk (mgr.14164) 174 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:53.583 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:53 smithi058 conmon[33537]: audit 2021-11-20T05:33:52.266426+0000 mon.smithi058 (mon.0) 442 : audit [DBG] from='client.? 172.21.15.58:0/1528051435' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-11-20T05:33:54.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.166+0000 7fb3b69a6700 1 -- 172.21.15.58:0/45924645 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb3b00f7960 msgr2=0x7fb3b00f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:54.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.166+0000 7fb3b69a6700 1 --2- 172.21.15.58:0/45924645 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb3b00f7960 0x7fb3b00f7dc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fb3a4004660 tx=0x7fb3a4009b30).stop 2021-11-20T05:33:54.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.166+0000 7fb3b69a6700 1 -- 172.21.15.58:0/45924645 shutdown_connections 2021-11-20T05:33:54.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.166+0000 7fb3b69a6700 1 --2- 172.21.15.58:0/45924645 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb3b00f7960 0x7fb3b00f7dc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:54.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.166+0000 7fb3b69a6700 1 --2- 172.21.15.58:0/45924645 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb3b00f66c0 0x7fb3b00f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:54.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.166+0000 7fb3b69a6700 1 -- 172.21.15.58:0/45924645 >> 172.21.15.58:0/45924645 conn(0x7fb3b00f1e20 msgr2=0x7fb3b00f4240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:54.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.166+0000 7fb3b69a6700 1 -- 172.21.15.58:0/45924645 shutdown_connections 2021-11-20T05:33:54.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.166+0000 7fb3b69a6700 1 -- 172.21.15.58:0/45924645 wait complete. 2021-11-20T05:33:54.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.166+0000 7fb3b69a6700 1 Processor -- start 2021-11-20T05:33:54.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.167+0000 7fb3b69a6700 1 -- start start 2021-11-20T05:33:54.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.167+0000 7fb3b69a6700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb3b00f66c0 0x7fb3b006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:54.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.167+0000 7fb3b69a6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb3b00f7960 0x7fb3b0069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:54.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.167+0000 7fb3b69a6700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b006a380 con 0x7fb3b00f66c0 2021-11-20T05:33:54.172 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.167+0000 7fb3b69a6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b006a4c0 con 0x7fb3b00f7960 2021-11-20T05:33:54.172 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.167+0000 7fb3affff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb3b00f66c0 0x7fb3b006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:54.172 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.167+0000 7fb3affff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb3b00f66c0 0x7fb3b006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40422/0 (socket says 172.21.15.58:40422) 2021-11-20T05:33:54.172 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.167+0000 7fb3affff700 1 -- 172.21.15.58:0/771050429 learned_addr learned my addr 172.21.15.58:0/771050429 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:54.173 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.167+0000 7fb3af7fe700 1 --2- 172.21.15.58:0/771050429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb3b00f7960 0x7fb3b0069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:54.173 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.167+0000 7fb3affff700 1 -- 172.21.15.58:0/771050429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb3b00f7960 msgr2=0x7fb3b0069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:54.173 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.167+0000 7fb3affff700 1 --2- 172.21.15.58:0/771050429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb3b00f7960 0x7fb3b0069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:54.173 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.167+0000 7fb3affff700 1 -- 172.21.15.58:0/771050429 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a4005040 con 0x7fb3b00f66c0 2021-11-20T05:33:54.173 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.167+0000 7fb3affff700 1 --2- 172.21.15.58:0/771050429 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb3b00f66c0 0x7fb3b006b9a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fb3a000c790 tx=0x7fb3a0007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:54.174 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.167+0000 7fb3acff9700 1 -- 172.21.15.58:0/771050429 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3a0013630 con 0x7fb3b00f66c0 2021-11-20T05:33:54.174 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.167+0000 7fb3acff9700 1 -- 172.21.15.58:0/771050429 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3a0012860 con 0x7fb3b00f66c0 2021-11-20T05:33:54.174 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.168+0000 7fb3acff9700 1 -- 172.21.15.58:0/771050429 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3a00027e0 con 0x7fb3b00f66c0 2021-11-20T05:33:54.174 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.168+0000 7fb3b69a6700 1 -- 172.21.15.58:0/771050429 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b006a7a0 con 0x7fb3b00f66c0 2021-11-20T05:33:54.175 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.168+0000 7fb3b69a6700 1 -- 172.21.15.58:0/771050429 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b010efa0 con 0x7fb3b00f66c0 2021-11-20T05:33:54.176 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.169+0000 7fb3b69a6700 1 -- 172.21.15.58:0/771050429 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b0045ad0 con 0x7fb3b00f66c0 2021-11-20T05:33:54.176 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.169+0000 7fb3acff9700 1 -- 172.21.15.58:0/771050429 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb3a0010070 con 0x7fb3b00f66c0 2021-11-20T05:33:54.176 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.169+0000 7fb3acff9700 1 --2- 172.21.15.58:0/771050429 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb39805c2b0 0x7fb39805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:54.177 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.169+0000 7fb3acff9700 1 -- 172.21.15.58:0/771050429 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fb3a0075c40 con 0x7fb3b00f66c0 2021-11-20T05:33:54.177 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.169+0000 7fb3af7fe700 1 --2- 172.21.15.58:0/771050429 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb39805c2b0 0x7fb39805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:54.177 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.170+0000 7fb3af7fe700 1 --2- 172.21.15.58:0/771050429 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb39805c2b0 0x7fb39805e770 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fb3a4004630 tx=0x7fb3a4009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:54.177 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.173+0000 7fb3acff9700 1 -- 172.21.15.58:0/771050429 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3a0048be0 con 0x7fb3b00f66c0 2021-11-20T05:33:54.367 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.366+0000 7fb3b69a6700 1 -- 172.21.15.58:0/771050429 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3b00fef90 con 0x7fb39805c2b0 2021-11-20T05:33:54.368 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.367+0000 7fb3acff9700 1 -- 172.21.15.58:0/771050429 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fb3b00fef90 con 0x7fb39805c2b0 2021-11-20T05:33:54.368 INFO:teuthology.orchestra.run.smithi058.stdout:Backend: cephadm 2021-11-20T05:33:54.369 INFO:teuthology.orchestra.run.smithi058.stdout:Available: Yes 2021-11-20T05:33:54.369 INFO:teuthology.orchestra.run.smithi058.stdout:Paused: No 2021-11-20T05:33:54.370 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.368+0000 7fb3b69a6700 1 -- 172.21.15.58:0/771050429 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb39805c2b0 msgr2=0x7fb39805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:54.370 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.369+0000 7fb3b69a6700 1 --2- 172.21.15.58:0/771050429 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb39805c2b0 0x7fb39805e770 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fb3a4004630 tx=0x7fb3a4009b30).stop 2021-11-20T05:33:54.370 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.369+0000 7fb3b69a6700 1 -- 172.21.15.58:0/771050429 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb3b00f66c0 msgr2=0x7fb3b006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:54.370 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.369+0000 7fb3b69a6700 1 --2- 172.21.15.58:0/771050429 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb3b00f66c0 0x7fb3b006b9a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fb3a000c790 tx=0x7fb3a0007d90).stop 2021-11-20T05:33:54.371 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.369+0000 7fb3b69a6700 1 -- 172.21.15.58:0/771050429 shutdown_connections 2021-11-20T05:33:54.372 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.369+0000 7fb3b69a6700 1 --2- 172.21.15.58:0/771050429 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb39805c2b0 0x7fb39805e770 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:54.372 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.369+0000 7fb3b69a6700 1 --2- 172.21.15.58:0/771050429 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb3b00f66c0 0x7fb3b006b9a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:54.372 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.369+0000 7fb3b69a6700 1 --2- 172.21.15.58:0/771050429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb3b00f7960 0x7fb3b0069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:54.373 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.369+0000 7fb3b69a6700 1 -- 172.21.15.58:0/771050429 >> 172.21.15.58:0/771050429 conn(0x7fb3b00f1e20 msgr2=0x7fb3b00fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:54.373 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.369+0000 7fb3b69a6700 1 -- 172.21.15.58:0/771050429 shutdown_connections 2021-11-20T05:33:54.373 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:54.369+0000 7fb3b69a6700 1 -- 172.21.15.58:0/771050429 wait complete. 2021-11-20T05:33:54.696 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch ps' 2021-11-20T05:33:55.444 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:55 smithi058 conmon[33537]: cluster 2021-11-20T05:33:54.206700+0000 mgr.smithi058.zagdlk (mgr.14164) 175 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:55.444 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:55 smithi058 conmon[33537]: audit 2021-11-20T05:33:54.368998+0000 mgr.smithi058.zagdlk (mgr.14164) 176 : audit [DBG] from='client.14400 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:33:55.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:55 smithi187 conmon[36358]: cluster 2021-11-20T05:33:54.206700+0000 mgr.smithi058.zagdlk (mgr.14164) 175 : cluster 2021-11-20T05:33:55.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:55 smithi187 conmon[36358]: [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:55.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:55 smithi187 conmon[36358]: audit 2021-11-20T05:33:54.368998+0000 mgr.smithi058.zagdlk (mgr.14164) 176 : audit [DBG] from='client.14400 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:33:56.180 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.178+0000 7f19dfc99700 1 -- 172.21.15.58:0/1743148633 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f19d80f9880 msgr2=0x7f19d80fbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:56.181 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.178+0000 7f19dfc99700 1 --2- 172.21.15.58:0/1743148633 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f19d80f9880 0x7f19d80fbc70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f19cc004660 tx=0x7f19cc009b30).stop 2021-11-20T05:33:56.181 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.178+0000 7f19dfc99700 1 -- 172.21.15.58:0/1743148633 shutdown_connections 2021-11-20T05:33:56.181 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.178+0000 7f19dfc99700 1 --2- 172.21.15.58:0/1743148633 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f19d80f9880 0x7f19d80fbc70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:56.181 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.178+0000 7f19dfc99700 1 --2- 172.21.15.58:0/1743148633 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19d80602b0 0x7f19d80f9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:56.182 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.178+0000 7f19dfc99700 1 -- 172.21.15.58:0/1743148633 >> 172.21.15.58:0/1743148633 conn(0x7f19d80f0d40 msgr2=0x7f19d80f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:56.182 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.178+0000 7f19dfc99700 1 -- 172.21.15.58:0/1743148633 shutdown_connections 2021-11-20T05:33:56.182 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.178+0000 7f19dfc99700 1 -- 172.21.15.58:0/1743148633 wait complete. 2021-11-20T05:33:56.183 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.178+0000 7f19dfc99700 1 Processor -- start 2021-11-20T05:33:56.183 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.178+0000 7f19dfc99700 1 -- start start 2021-11-20T05:33:56.183 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.179+0000 7f19dfc99700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19d80602b0 0x7f19d8106f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:56.183 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.179+0000 7f19dfc99700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f19d80f9880 0x7f19d8107440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:56.183 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.179+0000 7f19dfc99700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d81079f0 con 0x7f19d80f9880 2021-11-20T05:33:56.184 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.179+0000 7f19dfc99700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d8107b30 con 0x7f19d80602b0 2021-11-20T05:33:56.184 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.179+0000 7f19dd234700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f19d80f9880 0x7f19d8107440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:56.184 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.179+0000 7f19dd234700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f19d80f9880 0x7f19d8107440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40434/0 (socket says 172.21.15.58:40434) 2021-11-20T05:33:56.184 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.179+0000 7f19dd234700 1 -- 172.21.15.58:0/4210326567 learned_addr learned my addr 172.21.15.58:0/4210326567 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:56.185 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.179+0000 7f19dda35700 1 --2- 172.21.15.58:0/4210326567 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19d80602b0 0x7f19d8106f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:56.185 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.179+0000 7f19dd234700 1 -- 172.21.15.58:0/4210326567 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19d80602b0 msgr2=0x7f19d8106f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:56.185 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.179+0000 7f19dd234700 1 --2- 172.21.15.58:0/4210326567 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19d80602b0 0x7f19d8106f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:56.185 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.179+0000 7f19dd234700 1 -- 172.21.15.58:0/4210326567 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc005040 con 0x7f19d80f9880 2021-11-20T05:33:56.185 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.179+0000 7f19dd234700 1 --2- 172.21.15.58:0/4210326567 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f19d80f9880 0x7f19d8107440 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f19cc004e30 tx=0x7f19cc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:56.186 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.180+0000 7f19d67fc700 1 -- 172.21.15.58:0/4210326567 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19cc018070 con 0x7f19d80f9880 2021-11-20T05:33:56.186 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.180+0000 7f19dfc99700 1 -- 172.21.15.58:0/4210326567 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d8108470 con 0x7f19d80f9880 2021-11-20T05:33:56.187 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.180+0000 7f19d67fc700 1 -- 172.21.15.58:0/4210326567 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19cc013410 con 0x7f19d80f9880 2021-11-20T05:33:56.187 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.180+0000 7f19dfc99700 1 -- 172.21.15.58:0/4210326567 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d81089d0 con 0x7f19d80f9880 2021-11-20T05:33:56.188 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.180+0000 7f19d67fc700 1 -- 172.21.15.58:0/4210326567 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19cc01de20 con 0x7f19d80f9880 2021-11-20T05:33:56.188 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.181+0000 7f19dfc99700 1 -- 172.21.15.58:0/4210326567 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d8045ad0 con 0x7f19d80f9880 2021-11-20T05:33:56.188 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.181+0000 7f19d67fc700 1 -- 172.21.15.58:0/4210326567 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f19cc01c500 con 0x7f19d80f9880 2021-11-20T05:33:56.188 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.181+0000 7f19d67fc700 1 --2- 172.21.15.58:0/4210326567 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f19c005c2b0 0x7f19c005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:56.189 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.181+0000 7f19d67fc700 1 -- 172.21.15.58:0/4210326567 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f19cc0777d0 con 0x7f19d80f9880 2021-11-20T05:33:56.189 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.181+0000 7f19dda35700 1 --2- 172.21.15.58:0/4210326567 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f19c005c2b0 0x7f19c005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:56.189 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.182+0000 7f19dda35700 1 --2- 172.21.15.58:0/4210326567 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f19c005c2b0 0x7f19c005e770 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f19d0007db0 tx=0x7f19d0009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:56.189 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.185+0000 7f19d67fc700 1 -- 172.21.15.58:0/4210326567 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f19cc04a7d0 con 0x7f19d80f9880 2021-11-20T05:33:56.373 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.372+0000 7f19dfc99700 1 -- 172.21.15.58:0/4210326567 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f19d80fef60 con 0x7f19c005c2b0 2021-11-20T05:33:56.383 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.382+0000 7f19d67fc700 1 -- 172.21.15.58:0/4210326567 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2160 (secure 0 0 0) 0x7f19d80fef60 con 0x7f19c005c2b0 2021-11-20T05:33:56.385 INFO:teuthology.orchestra.run.smithi058.stdout:NAME HOST PORTS STATUS REFRESHED AGE VERSION IMAGE ID CONTAINER ID 2021-11-20T05:33:56.385 INFO:teuthology.orchestra.run.smithi058.stdout:alertmanager.smithi058 smithi058 *:9093,9094 running (2m) 40s ago 3m 0.20.0 0881eb8f169f e5ed10264ece 2021-11-20T05:33:56.385 INFO:teuthology.orchestra.run.smithi058.stdout:crash.smithi058 smithi058 running (3m) 40s ago 3m 16.2.4 8d91d370c2b8 32abfda9c9a3 2021-11-20T05:33:56.386 INFO:teuthology.orchestra.run.smithi058.stdout:crash.smithi187 smithi187 running (3m) 50s ago 3m 16.2.4 8d91d370c2b8 fe7bf724a579 2021-11-20T05:33:56.386 INFO:teuthology.orchestra.run.smithi058.stdout:grafana.smithi058 smithi058 *:3000 running (2m) 40s ago 3m 6.7.4 557c83e11646 d1a988dec798 2021-11-20T05:33:56.386 INFO:teuthology.orchestra.run.smithi058.stdout:mgr.smithi058.zagdlk smithi058 *:9283 running (4m) 40s ago 4m 16.2.4 8d91d370c2b8 a15b42997b5d 2021-11-20T05:33:56.386 INFO:teuthology.orchestra.run.smithi058.stdout:mgr.smithi187.omkctn smithi187 *:8443,9283 running (2m) 50s ago 2m 16.2.4 8d91d370c2b8 c27122af7b5b 2021-11-20T05:33:56.386 INFO:teuthology.orchestra.run.smithi058.stdout:mon.smithi058 smithi058 running (4m) 40s ago 5m 16.2.4 8d91d370c2b8 cb3d93de393a 2021-11-20T05:33:56.387 INFO:teuthology.orchestra.run.smithi058.stdout:mon.smithi187 smithi187 running (2m) 50s ago 2m 16.2.4 8d91d370c2b8 6f61445461b0 2021-11-20T05:33:56.387 INFO:teuthology.orchestra.run.smithi058.stdout:node-exporter.smithi058 smithi058 *:9100 running (3m) 40s ago 3m 0.18.1 e5a616e4b9cf 0267356ce6cb 2021-11-20T05:33:56.387 INFO:teuthology.orchestra.run.smithi058.stdout:node-exporter.smithi187 smithi187 *:9100 running (2m) 50s ago 2m 0.18.1 e5a616e4b9cf 1cf4d398ce64 2021-11-20T05:33:56.387 INFO:teuthology.orchestra.run.smithi058.stdout:osd.0 smithi058 running (2m) 40s ago 2m 16.2.4 8d91d370c2b8 78ada5fa33eb 2021-11-20T05:33:56.388 INFO:teuthology.orchestra.run.smithi058.stdout:osd.1 smithi058 running (2m) 40s ago 2m 16.2.4 8d91d370c2b8 74a70fdeb465 2021-11-20T05:33:56.388 INFO:teuthology.orchestra.run.smithi058.stdout:osd.2 smithi058 running (106s) 40s ago 106s 16.2.4 8d91d370c2b8 ccac1f29d918 2021-11-20T05:33:56.388 INFO:teuthology.orchestra.run.smithi058.stdout:osd.3 smithi187 running (90s) 50s ago 89s 16.2.4 8d91d370c2b8 da8c6aa4d59b 2021-11-20T05:33:56.388 INFO:teuthology.orchestra.run.smithi058.stdout:osd.4 smithi187 running (71s) 50s ago 71s 16.2.4 8d91d370c2b8 28a8b0af2e17 2021-11-20T05:33:56.389 INFO:teuthology.orchestra.run.smithi058.stdout:osd.5 smithi187 running (53s) 50s ago 53s 16.2.4 8d91d370c2b8 89a766d980ac 2021-11-20T05:33:56.389 INFO:teuthology.orchestra.run.smithi058.stdout:prometheus.smithi058 smithi058 *:9095 running (2m) 40s ago 3m 2.18.1 de242295e225 94f750d32daa 2021-11-20T05:33:56.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.384+0000 7f19dfc99700 1 -- 172.21.15.58:0/4210326567 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f19c005c2b0 msgr2=0x7f19c005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:56.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.384+0000 7f19dfc99700 1 --2- 172.21.15.58:0/4210326567 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f19c005c2b0 0x7f19c005e770 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f19d0007db0 tx=0x7f19d0009040).stop 2021-11-20T05:33:56.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.384+0000 7f19dfc99700 1 -- 172.21.15.58:0/4210326567 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f19d80f9880 msgr2=0x7f19d8107440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:56.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.384+0000 7f19dfc99700 1 --2- 172.21.15.58:0/4210326567 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f19d80f9880 0x7f19d8107440 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f19cc004e30 tx=0x7f19cc009b30).stop 2021-11-20T05:33:56.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.384+0000 7f19dfc99700 1 -- 172.21.15.58:0/4210326567 shutdown_connections 2021-11-20T05:33:56.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.384+0000 7f19dfc99700 1 --2- 172.21.15.58:0/4210326567 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f19c005c2b0 0x7f19c005e770 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:56.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.384+0000 7f19dfc99700 1 --2- 172.21.15.58:0/4210326567 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19d80602b0 0x7f19d8106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:56.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.384+0000 7f19dfc99700 1 --2- 172.21.15.58:0/4210326567 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f19d80f9880 0x7f19d8107440 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:56.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.384+0000 7f19dfc99700 1 -- 172.21.15.58:0/4210326567 >> 172.21.15.58:0/4210326567 conn(0x7f19d80f0d40 msgr2=0x7f19d80f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:56.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.384+0000 7f19dfc99700 1 -- 172.21.15.58:0/4210326567 shutdown_connections 2021-11-20T05:33:56.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:56.384+0000 7f19dfc99700 1 -- 172.21.15.58:0/4210326567 wait complete. 2021-11-20T05:33:56.708 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch ls' 2021-11-20T05:33:57.402 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:57 smithi058 conmon[33537]: cluster 2021-11-20T05:33:56.207045+0000 mgr.smithi058.zagdlk (mgr.14164) 177 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:57.402 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:57 smithi058 conmon[33537]: audit 2021-11-20T05:33:56.374867+0000 mgr.smithi058.zagdlk (mgr.14164) 178 : audit [DBG] from='client.14404 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:33:57.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:57 smithi187 conmon[36358]: cluster 2021-11-20T05:33:56.207045+0000 mgr.smithi058.zagdlk (mgr.14164) 177 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:57.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:57 smithi187 conmon[36358]: audit 2021-11-20T05:33:56.374867+0000 mgr.smithi058.zagdlk (mgr.14164) 178 : audit [DBG] from='client.14404 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:33:58.162 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.159+0000 7f7955ec2700 1 -- 172.21.15.58:0/2394786247 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f79500f5ef0 msgr2=0x7f79500f6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:58.162 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.159+0000 7f7955ec2700 1 --2- 172.21.15.58:0/2394786247 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f79500f5ef0 0x7f79500f6920 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f7944004660 tx=0x7f7944009b30).stop 2021-11-20T05:33:58.163 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.159+0000 7f7955ec2700 1 -- 172.21.15.58:0/2394786247 shutdown_connections 2021-11-20T05:33:58.163 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.159+0000 7f7955ec2700 1 --2- 172.21.15.58:0/2394786247 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f79500f5ef0 0x7f79500f6920 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:58.163 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.159+0000 7f7955ec2700 1 --2- 172.21.15.58:0/2394786247 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79500f55e0 0x7f79500f5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:58.163 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.159+0000 7f7955ec2700 1 -- 172.21.15.58:0/2394786247 >> 172.21.15.58:0/2394786247 conn(0x7f79500f0de0 msgr2=0x7f79500f3200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:58.164 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.159+0000 7f7955ec2700 1 -- 172.21.15.58:0/2394786247 shutdown_connections 2021-11-20T05:33:58.164 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.159+0000 7f7955ec2700 1 -- 172.21.15.58:0/2394786247 wait complete. 2021-11-20T05:33:58.164 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.160+0000 7f7955ec2700 1 Processor -- start 2021-11-20T05:33:58.164 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.160+0000 7f7955ec2700 1 -- start start 2021-11-20T05:33:58.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.160+0000 7f7955ec2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79500f55e0 0x7f795010c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:58.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.160+0000 7f7955ec2700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f79500f5ef0 0x7f795010e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:58.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.160+0000 7f7955ec2700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79500603d0 con 0x7f79500f5ef0 2021-11-20T05:33:58.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.160+0000 7f7955ec2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7950060510 con 0x7f79500f55e0 2021-11-20T05:33:58.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.160+0000 7f794effd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f79500f5ef0 0x7f795010e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:58.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.160+0000 7f794effd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f79500f5ef0 0x7f795010e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40444/0 (socket says 172.21.15.58:40444) 2021-11-20T05:33:58.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.160+0000 7f794effd700 1 -- 172.21.15.58:0/2021452660 learned_addr learned my addr 172.21.15.58:0/2021452660 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:33:58.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.160+0000 7f794f7fe700 1 --2- 172.21.15.58:0/2021452660 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79500f55e0 0x7f795010c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:58.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.160+0000 7f794effd700 1 -- 172.21.15.58:0/2021452660 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79500f55e0 msgr2=0x7f795010c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:58.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.160+0000 7f794effd700 1 --2- 172.21.15.58:0/2021452660 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79500f55e0 0x7f795010c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:58.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.161+0000 7f794effd700 1 -- 172.21.15.58:0/2021452660 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7944005040 con 0x7f79500f5ef0 2021-11-20T05:33:58.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.161+0000 7f794effd700 1 --2- 172.21.15.58:0/2021452660 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f79500f5ef0 0x7f795010e530 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f7944009f50 tx=0x7f7944009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:58.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.161+0000 7f7937fff700 1 -- 172.21.15.58:0/2021452660 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7944018070 con 0x7f79500f5ef0 2021-11-20T05:33:58.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.161+0000 7f7955ec2700 1 -- 172.21.15.58:0/2021452660 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f795010ea20 con 0x7f79500f5ef0 2021-11-20T05:33:58.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.161+0000 7f7937fff700 1 -- 172.21.15.58:0/2021452660 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7944013410 con 0x7f79500f5ef0 2021-11-20T05:33:58.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.161+0000 7f7937fff700 1 -- 172.21.15.58:0/2021452660 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f794401c3c0 con 0x7f79500f5ef0 2021-11-20T05:33:58.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.161+0000 7f7955ec2700 1 -- 172.21.15.58:0/2021452660 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f795010edf0 con 0x7f79500f5ef0 2021-11-20T05:33:58.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.162+0000 7f7955ec2700 1 -- 172.21.15.58:0/2021452660 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7950045ad0 con 0x7f79500f5ef0 2021-11-20T05:33:58.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.162+0000 7f7937fff700 1 -- 172.21.15.58:0/2021452660 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f794401d420 con 0x7f79500f5ef0 2021-11-20T05:33:58.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.163+0000 7f7937fff700 1 --2- 172.21.15.58:0/2021452660 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f793805c2b0 0x7f793805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:33:58.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.163+0000 7f7937fff700 1 -- 172.21.15.58:0/2021452660 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7944077800 con 0x7f79500f5ef0 2021-11-20T05:33:58.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.163+0000 7f794f7fe700 1 --2- 172.21.15.58:0/2021452660 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f793805c2b0 0x7f793805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:33:58.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.163+0000 7f794f7fe700 1 --2- 172.21.15.58:0/2021452660 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f793805c2b0 0x7f793805e770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f7950060160 tx=0x7f794000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:33:58.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.168+0000 7f7937fff700 1 -- 172.21.15.58:0/2021452660 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f794404a800 con 0x7f79500f5ef0 2021-11-20T05:33:58.352 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.352+0000 7f7955ec2700 1 -- 172.21.15.58:0/2021452660 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7950057f10 con 0x7f793805c2b0 2021-11-20T05:33:58.357 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.358+0000 7f7937fff700 1 -- 172.21.15.58:0/2021452660 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1116 (secure 0 0 0) 0x7f7950057f10 con 0x7f793805c2b0 2021-11-20T05:33:58.358 INFO:teuthology.orchestra.run.smithi058.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2021-11-20T05:33:58.358 INFO:teuthology.orchestra.run.smithi058.stdout:alertmanager 1/1 42s ago 4m count:1 2021-11-20T05:33:58.358 INFO:teuthology.orchestra.run.smithi058.stdout:crash 2/2 52s ago 4m * 2021-11-20T05:33:58.359 INFO:teuthology.orchestra.run.smithi058.stdout:grafana 1/1 42s ago 4m count:1 2021-11-20T05:33:58.359 INFO:teuthology.orchestra.run.smithi058.stdout:mgr 2/2 52s ago 4m count:2 2021-11-20T05:33:58.359 INFO:teuthology.orchestra.run.smithi058.stdout:mon 2/2 52s ago 3m smithi058:172.21.15.58=smithi058;smithi187:172.21.15.187=smithi187;count:2 2021-11-20T05:33:58.359 INFO:teuthology.orchestra.run.smithi058.stdout:node-exporter 2/2 52s ago 4m * 2021-11-20T05:33:58.359 INFO:teuthology.orchestra.run.smithi058.stdout:osd.unmanaged 6/6 52s ago - 2021-11-20T05:33:58.360 INFO:teuthology.orchestra.run.smithi058.stdout:prometheus 1/1 42s ago 4m count:1 2021-11-20T05:33:58.361 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.359+0000 7f7955ec2700 1 -- 172.21.15.58:0/2021452660 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f793805c2b0 msgr2=0x7f793805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:58.361 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.359+0000 7f7955ec2700 1 --2- 172.21.15.58:0/2021452660 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f793805c2b0 0x7f793805e770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f7950060160 tx=0x7f794000b040).stop 2021-11-20T05:33:58.362 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.359+0000 7f7955ec2700 1 -- 172.21.15.58:0/2021452660 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f79500f5ef0 msgr2=0x7f795010e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:33:58.362 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.359+0000 7f7955ec2700 1 --2- 172.21.15.58:0/2021452660 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f79500f5ef0 0x7f795010e530 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f7944009f50 tx=0x7f7944009b30).stop 2021-11-20T05:33:58.362 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.359+0000 7f7955ec2700 1 -- 172.21.15.58:0/2021452660 shutdown_connections 2021-11-20T05:33:58.362 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.359+0000 7f7955ec2700 1 --2- 172.21.15.58:0/2021452660 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f793805c2b0 0x7f793805e770 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:58.363 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.359+0000 7f7955ec2700 1 --2- 172.21.15.58:0/2021452660 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79500f55e0 0x7f795010c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:58.363 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.359+0000 7f7955ec2700 1 --2- 172.21.15.58:0/2021452660 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f79500f5ef0 0x7f795010e530 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:33:58.363 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.359+0000 7f7955ec2700 1 -- 172.21.15.58:0/2021452660 >> 172.21.15.58:0/2021452660 conn(0x7f79500f0de0 msgr2=0x7f79500fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:33:58.363 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.359+0000 7f7955ec2700 1 -- 172.21.15.58:0/2021452660 shutdown_connections 2021-11-20T05:33:58.364 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:33:58.359+0000 7f7955ec2700 1 -- 172.21.15.58:0/2021452660 wait complete. 2021-11-20T05:33:58.736 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch host ls' 2021-11-20T05:33:59.418 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:59 smithi058 conmon[33537]: cluster 2021-11-20T05:33:58.207367+0000 mgr.smithi058.zagdlk (mgr.14164) 179 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:59.419 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:33:59 smithi058 conmon[33537]: audit 2021-11-20T05:33:58.354753+0000 mgr.smithi058.zagdlk (mgr.14164) 180 : audit [DBG] from='client.14408 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:33:59.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:59 smithi187 conmon[36358]: cluster 2021-11-20T05:33:58.207367+0000 mgr.smithi058.zagdlk (mgr.14164) 179 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:33:59.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:33:59 smithi187 conmon[36358]: audit 2021-11-20T05:33:58.354753+0000 mgr.smithi058.zagdlk (mgr.14164) 180 : audit [DBG] from='client.14408 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:34:00.256 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.254+0000 7fc5edccb700 1 -- 172.21.15.58:0/1349903113 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc5e80f5440 msgr2=0x7fc5e80f5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:00.257 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.254+0000 7fc5edccb700 1 --2- 172.21.15.58:0/1349903113 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc5e80f5440 0x7fc5e80f5860 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fc5dc004660 tx=0x7fc5dc009b30).stop 2021-11-20T05:34:00.257 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.254+0000 7fc5edccb700 1 -- 172.21.15.58:0/1349903113 shutdown_connections 2021-11-20T05:34:00.257 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.254+0000 7fc5edccb700 1 --2- 172.21.15.58:0/1349903113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5e80f66e0 0x7fc5e80f6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:00.257 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.254+0000 7fc5edccb700 1 --2- 172.21.15.58:0/1349903113 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc5e80f5440 0x7fc5e80f5860 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:00.258 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.254+0000 7fc5edccb700 1 -- 172.21.15.58:0/1349903113 >> 172.21.15.58:0/1349903113 conn(0x7fc5e80f0ba0 msgr2=0x7fc5e80f2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:00.258 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.254+0000 7fc5edccb700 1 -- 172.21.15.58:0/1349903113 shutdown_connections 2021-11-20T05:34:00.258 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.254+0000 7fc5edccb700 1 -- 172.21.15.58:0/1349903113 wait complete. 2021-11-20T05:34:00.258 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.255+0000 7fc5edccb700 1 Processor -- start 2021-11-20T05:34:00.259 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.255+0000 7fc5edccb700 1 -- start start 2021-11-20T05:34:00.259 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.255+0000 7fc5edccb700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc5e80f5440 0x7fc5e80f95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:00.259 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.255+0000 7fc5edccb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5e80f66e0 0x7fc5e80f7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:00.260 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.255+0000 7fc5edccb700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e80f7f80 con 0x7fc5e80f5440 2021-11-20T05:34:00.260 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.255+0000 7fc5edccb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e80f80c0 con 0x7fc5e80f66e0 2021-11-20T05:34:00.260 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.255+0000 7fc5eccc9700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc5e80f5440 0x7fc5e80f95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:00.260 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.255+0000 7fc5eccc9700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc5e80f5440 0x7fc5e80f95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40452/0 (socket says 172.21.15.58:40452) 2021-11-20T05:34:00.261 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.255+0000 7fc5eccc9700 1 -- 172.21.15.58:0/159751590 learned_addr learned my addr 172.21.15.58:0/159751590 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:00.261 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.256+0000 7fc5e7fff700 1 --2- 172.21.15.58:0/159751590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5e80f66e0 0x7fc5e80f7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:00.261 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.256+0000 7fc5eccc9700 1 -- 172.21.15.58:0/159751590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5e80f66e0 msgr2=0x7fc5e80f7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:00.261 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.256+0000 7fc5eccc9700 1 --2- 172.21.15.58:0/159751590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5e80f66e0 0x7fc5e80f7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:00.262 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.256+0000 7fc5eccc9700 1 -- 172.21.15.58:0/159751590 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5dc005040 con 0x7fc5e80f5440 2021-11-20T05:34:00.262 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.256+0000 7fc5eccc9700 1 --2- 172.21.15.58:0/159751590 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc5e80f5440 0x7fc5e80f95a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fc5dc004e30 tx=0x7fc5dc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:00.262 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.256+0000 7fc5e57fa700 1 -- 172.21.15.58:0/159751590 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5dc018070 con 0x7fc5e80f5440 2021-11-20T05:34:00.262 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.256+0000 7fc5e57fa700 1 -- 172.21.15.58:0/159751590 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5dc013410 con 0x7fc5e80f5440 2021-11-20T05:34:00.262 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.256+0000 7fc5edccb700 1 -- 172.21.15.58:0/159751590 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e80f8340 con 0x7fc5e80f5440 2021-11-20T05:34:00.263 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.256+0000 7fc5e57fa700 1 -- 172.21.15.58:0/159751590 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5dc01c3c0 con 0x7fc5e80f5440 2021-11-20T05:34:00.263 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.256+0000 7fc5edccb700 1 -- 172.21.15.58:0/159751590 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e80f88a0 con 0x7fc5e80f5440 2021-11-20T05:34:00.263 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.257+0000 7fc5edccb700 1 -- 172.21.15.58:0/159751590 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e8045ad0 con 0x7fc5e80f5440 2021-11-20T05:34:00.264 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.257+0000 7fc5e57fa700 1 -- 172.21.15.58:0/159751590 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc5dc01d420 con 0x7fc5e80f5440 2021-11-20T05:34:00.265 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.258+0000 7fc5e57fa700 1 --2- 172.21.15.58:0/159751590 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc5d805c300 0x7fc5d805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:00.265 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.258+0000 7fc5e57fa700 1 -- 172.21.15.58:0/159751590 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fc5dc077820 con 0x7fc5e80f5440 2021-11-20T05:34:00.265 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.258+0000 7fc5e7fff700 1 --2- 172.21.15.58:0/159751590 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc5d805c300 0x7fc5d805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:00.266 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.258+0000 7fc5e7fff700 1 --2- 172.21.15.58:0/159751590 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc5d805c300 0x7fc5d805e7c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fc5d4009760 tx=0x7fc5d4009c60).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:00.266 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.262+0000 7fc5e57fa700 1 -- 172.21.15.58:0/159751590 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc5dc04a820 con 0x7fc5e80f5440 2021-11-20T05:34:00.445 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.445+0000 7fc5edccb700 1 -- 172.21.15.58:0/159751590 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5e80fab20 con 0x7fc5d805c300 2021-11-20T05:34:00.446 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.446+0000 7fc5e57fa700 1 -- 172.21.15.58:0/159751590 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+126 (secure 0 0 0) 0x7fc5e80fab20 con 0x7fc5d805c300 2021-11-20T05:34:00.446 INFO:teuthology.orchestra.run.smithi058.stdout:HOST ADDR LABELS STATUS 2021-11-20T05:34:00.447 INFO:teuthology.orchestra.run.smithi058.stdout:smithi058 172.21.15.58 2021-11-20T05:34:00.447 INFO:teuthology.orchestra.run.smithi058.stdout:smithi187 smithi187 2021-11-20T05:34:00.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.448+0000 7fc5edccb700 1 -- 172.21.15.58:0/159751590 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc5d805c300 msgr2=0x7fc5d805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:00.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.448+0000 7fc5edccb700 1 --2- 172.21.15.58:0/159751590 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc5d805c300 0x7fc5d805e7c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fc5d4009760 tx=0x7fc5d4009c60).stop 2021-11-20T05:34:00.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.448+0000 7fc5edccb700 1 -- 172.21.15.58:0/159751590 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc5e80f5440 msgr2=0x7fc5e80f95a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:00.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.448+0000 7fc5edccb700 1 --2- 172.21.15.58:0/159751590 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc5e80f5440 0x7fc5e80f95a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fc5dc004e30 tx=0x7fc5dc009b30).stop 2021-11-20T05:34:00.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.448+0000 7fc5edccb700 1 -- 172.21.15.58:0/159751590 shutdown_connections 2021-11-20T05:34:00.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.448+0000 7fc5edccb700 1 --2- 172.21.15.58:0/159751590 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc5d805c300 0x7fc5d805e7c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:00.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.448+0000 7fc5edccb700 1 --2- 172.21.15.58:0/159751590 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc5e80f5440 0x7fc5e80f95a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:00.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.448+0000 7fc5edccb700 1 --2- 172.21.15.58:0/159751590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5e80f66e0 0x7fc5e80f7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:00.450 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.448+0000 7fc5edccb700 1 -- 172.21.15.58:0/159751590 >> 172.21.15.58:0/159751590 conn(0x7fc5e80f0ba0 msgr2=0x7fc5e805fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:00.450 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.448+0000 7fc5edccb700 1 -- 172.21.15.58:0/159751590 shutdown_connections 2021-11-20T05:34:00.450 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:00.448+0000 7fc5edccb700 1 -- 172.21.15.58:0/159751590 wait complete. 2021-11-20T05:34:00.862 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch device ls' 2021-11-20T05:34:01.527 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:01 smithi058 conmon[33537]: cluster 2021-11-20T05:34:00.207624+0000 mgr.smithi058.zagdlk (mgr.14164) 181 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:01.528 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:01 smithi058 conmon[33537]: audit 2021-11-20T05:34:00.373949+0000 mon.smithi058 (mon.0) 443 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:34:01.528 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:01 smithi058 conmon[33537]: audit 2021-11-20T05:34:00.374385+0000 mon.smithi058 (mon.0) 444 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:34:01.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:01 smithi187 conmon[36358]: cluster 2021-11-20T05:34:00.207624+0000 mgr.smithi058.zagdlk (mgr.14164) 181 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:01.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:01 smithi187 conmon[36358]: audit 2021-11-20T05:34: 2021-11-20T05:34:01.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:01 smithi187 conmon[36358]: 00.373949+0000 mon.smithi058 (mon.0) 443 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:34:01.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:01 smithi187 conmon[36358]: audit 2021-11-20T05:34:00.374385+0000 mon.smithi058 (mon.0) 444 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:34:02.252 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.251+0000 7fc13eddb700 1 -- 172.21.15.58:0/3798385382 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1380f55c0 msgr2=0x7fc1380f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:02.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.251+0000 7fc13eddb700 1 --2- 172.21.15.58:0/3798385382 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1380f55c0 0x7fc1380f59e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fc1300099e0 tx=0x7fc130004d10).stop 2021-11-20T05:34:02.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.251+0000 7fc13eddb700 1 -- 172.21.15.58:0/3798385382 shutdown_connections 2021-11-20T05:34:02.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.251+0000 7fc13eddb700 1 --2- 172.21.15.58:0/3798385382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1380f6860 0x7fc1380f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:02.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.251+0000 7fc13eddb700 1 --2- 172.21.15.58:0/3798385382 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1380f55c0 0x7fc1380f59e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:02.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.251+0000 7fc13eddb700 1 -- 172.21.15.58:0/3798385382 >> 172.21.15.58:0/3798385382 conn(0x7fc1380f0d40 msgr2=0x7fc1380f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:02.254 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.251+0000 7fc13eddb700 1 -- 172.21.15.58:0/3798385382 shutdown_connections 2021-11-20T05:34:02.254 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.251+0000 7fc13eddb700 1 -- 172.21.15.58:0/3798385382 wait complete. 2021-11-20T05:34:02.254 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.251+0000 7fc13eddb700 1 Processor -- start 2021-11-20T05:34:02.254 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.251+0000 7fc13eddb700 1 -- start start 2021-11-20T05:34:02.255 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.252+0000 7fc13eddb700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1380f55c0 0x7fc138107080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:02.255 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.252+0000 7fc13eddb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1380f6860 0x7fc138107570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:02.255 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.252+0000 7fc13eddb700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc138107b20 con 0x7fc1380f55c0 2021-11-20T05:34:02.255 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.252+0000 7fc13eddb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc138107c60 con 0x7fc1380f6860 2021-11-20T05:34:02.256 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.252+0000 7fc13cb77700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1380f55c0 0x7fc138107080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:02.256 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.252+0000 7fc13cb77700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1380f55c0 0x7fc138107080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40462/0 (socket says 172.21.15.58:40462) 2021-11-20T05:34:02.256 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.252+0000 7fc13cb77700 1 -- 172.21.15.58:0/782957679 learned_addr learned my addr 172.21.15.58:0/782957679 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:02.256 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.252+0000 7fc137fff700 1 --2- 172.21.15.58:0/782957679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1380f6860 0x7fc138107570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:02.257 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.252+0000 7fc13cb77700 1 -- 172.21.15.58:0/782957679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1380f6860 msgr2=0x7fc138107570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:02.257 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.252+0000 7fc13cb77700 1 --2- 172.21.15.58:0/782957679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1380f6860 0x7fc138107570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:02.257 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.252+0000 7fc13cb77700 1 -- 172.21.15.58:0/782957679 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc130009640 con 0x7fc1380f55c0 2021-11-20T05:34:02.257 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.252+0000 7fc13cb77700 1 --2- 172.21.15.58:0/782957679 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1380f55c0 0x7fc138107080 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fc130007830 tx=0x7fc130004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:02.257 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.252+0000 7fc1357fa700 1 -- 172.21.15.58:0/782957679 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc13000e030 con 0x7fc1380f55c0 2021-11-20T05:34:02.258 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.252+0000 7fc1357fa700 1 -- 172.21.15.58:0/782957679 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc13001d8d0 con 0x7fc1380f55c0 2021-11-20T05:34:02.258 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.252+0000 7fc13eddb700 1 -- 172.21.15.58:0/782957679 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1381085a0 con 0x7fc1380f55c0 2021-11-20T05:34:02.258 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.252+0000 7fc1357fa700 1 -- 172.21.15.58:0/782957679 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc130018070 con 0x7fc1380f55c0 2021-11-20T05:34:02.258 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.252+0000 7fc13eddb700 1 -- 172.21.15.58:0/782957679 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc138108a80 con 0x7fc1380f55c0 2021-11-20T05:34:02.259 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.253+0000 7fc13eddb700 1 -- 172.21.15.58:0/782957679 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc138045ad0 con 0x7fc1380f55c0 2021-11-20T05:34:02.260 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.254+0000 7fc1357fa700 1 -- 172.21.15.58:0/782957679 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc13001d420 con 0x7fc1380f55c0 2021-11-20T05:34:02.260 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.254+0000 7fc1357fa700 1 --2- 172.21.15.58:0/782957679 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc12005c2b0 0x7fc12005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:02.260 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.254+0000 7fc1357fa700 1 -- 172.21.15.58:0/782957679 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fc1300779d0 con 0x7fc1380f55c0 2021-11-20T05:34:02.261 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.254+0000 7fc137fff700 1 --2- 172.21.15.58:0/782957679 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc12005c2b0 0x7fc12005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:02.261 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.255+0000 7fc137fff700 1 --2- 172.21.15.58:0/782957679 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc12005c2b0 0x7fc12005e770 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fc12c007fd0 tx=0x7fc12c009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:02.261 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.258+0000 7fc1357fa700 1 -- 172.21.15.58:0/782957679 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc13004aa50 con 0x7fc1380f55c0 2021-11-20T05:34:02.442 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.443+0000 7fc13eddb700 1 -- 172.21.15.58:0/782957679 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc138109280 con 0x7fc12005c2b0 2021-11-20T05:34:02.444 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.444+0000 7fc1357fa700 1 -- 172.21.15.58:0/782957679 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+279 (secure 0 0 0) 0x7fc138109280 con 0x7fc12005c2b0 2021-11-20T05:34:02.444 INFO:teuthology.orchestra.run.smithi058.stdout:Hostname Path Type Serial Size Health Ident Fault Available 2021-11-20T05:34:02.444 INFO:teuthology.orchestra.run.smithi058.stdout:smithi058 /dev/nvme0n1 ssd PHMB751300GN480DGN 480G Unknown N/A N/A No 2021-11-20T05:34:02.445 INFO:teuthology.orchestra.run.smithi058.stdout:smithi187 /dev/nvme0n1 ssd PHFT620500MW400BGN 400G Unknown N/A N/A No 2021-11-20T05:34:02.446 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.446+0000 7fc13eddb700 1 -- 172.21.15.58:0/782957679 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc12005c2b0 msgr2=0x7fc12005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:02.446 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.446+0000 7fc13eddb700 1 --2- 172.21.15.58:0/782957679 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc12005c2b0 0x7fc12005e770 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fc12c007fd0 tx=0x7fc12c009040).stop 2021-11-20T05:34:02.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.446+0000 7fc13eddb700 1 -- 172.21.15.58:0/782957679 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1380f55c0 msgr2=0x7fc138107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:02.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.446+0000 7fc13eddb700 1 --2- 172.21.15.58:0/782957679 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1380f55c0 0x7fc138107080 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fc130007830 tx=0x7fc130004d10).stop 2021-11-20T05:34:02.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.446+0000 7fc13eddb700 1 -- 172.21.15.58:0/782957679 shutdown_connections 2021-11-20T05:34:02.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.446+0000 7fc13eddb700 1 --2- 172.21.15.58:0/782957679 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc12005c2b0 0x7fc12005e770 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:02.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.446+0000 7fc13eddb700 1 --2- 172.21.15.58:0/782957679 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1380f55c0 0x7fc138107080 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:02.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.446+0000 7fc13eddb700 1 --2- 172.21.15.58:0/782957679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1380f6860 0x7fc138107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:02.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.446+0000 7fc13eddb700 1 -- 172.21.15.58:0/782957679 >> 172.21.15.58:0/782957679 conn(0x7fc1380f0d40 msgr2=0x7fc13805fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:02.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.446+0000 7fc13eddb700 1 -- 172.21.15.58:0/782957679 shutdown_connections 2021-11-20T05:34:02.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:02.446+0000 7fc13eddb700 1 -- 172.21.15.58:0/782957679 wait complete. 2021-11-20T05:34:02.631 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:02 smithi058 conmon[33537]: audit 2021-11-20T05 2021-11-20T05:34:02.631 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:02 smithi058 conmon[33537]: :34:00.446970+0000 mgr.smithi058.zagdlk (mgr.14164) 182 : audit [DBG] from='client.14412 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:34:02.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:02 smithi187 conmon[36358]: audit 2021-11-20T05:34:00.446970+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:34:02.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:02 smithi187 conmon[36358]: 182 : audit [DBG] from='client.14412 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:34:02.788 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T05:34:02.800 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi058.front.sepia.ceph.com 2021-11-20T05:34:02.801 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs volume create cephfs --placement=4' 2021-11-20T05:34:03.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:03 smithi187 conmon[36358]: cluster 2021-11-20T05:34:02.207980+0000 mgr.smithi058.zagdlk (mgr.14164) 183 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:03.597 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:03 smithi058 conmon[33537]: cluster 2021-11-20T05:34:02 2021-11-20T05:34:03.598 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:03 smithi058 conmon[33537]: .207980+0000 mgr.smithi058.zagdlk (mgr.14164) 183 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:04.147 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.146+0000 7fc6b0ceb700 1 -- 172.21.15.58:0/464639241 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc6ac0f55e0 msgr2=0x7fc6ac0f5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:04.148 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.146+0000 7fc6b0ceb700 1 --2- 172.21.15.58:0/464639241 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc6ac0f55e0 0x7fc6ac0f5a00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fc69c004660 tx=0x7fc69c009b30).stop 2021-11-20T05:34:04.148 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.146+0000 7fc6b0ceb700 1 -- 172.21.15.58:0/464639241 shutdown_connections 2021-11-20T05:34:04.148 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.146+0000 7fc6b0ceb700 1 --2- 172.21.15.58:0/464639241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6ac0f5ef0 0x7fc6ac0f6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:34:04.149 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.146+0000 7fc6b0ceb700 1 --2- 172.21.15.58:0/464639241 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc6ac0f55e0 0x7fc6ac0f5a00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:04.149 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.146+0000 7fc6b0ceb700 1 -- 172.21.15.58:0/464639241 >> 172.21.15.58:0/464639241 conn(0x7fc6ac0f0de0 msgr2=0x7fc6ac0f3200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:04.149 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.146+0000 7fc6b0ceb700 1 -- 172.21.15.58:0/464639241 shutdown_connections 2021-11-20T05:34:04.149 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.146+0000 7fc6b0ceb700 1 -- 172.21.15.58:0/464639241 wait complete. 2021-11-20T05:34:04.149 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.147+0000 7fc6b0ceb700 1 Processor -- start 2021-11-20T05:34:04.150 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.147+0000 7fc6b0ceb700 1 -- start start 2021-11-20T05:34:04.150 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.147+0000 7fc6b0ceb700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc6ac0f55e0 0x7fc6ac10c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:04.150 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.147+0000 7fc6b0ceb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6ac0f5ef0 0x7fc6ac10e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:04.150 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.147+0000 7fc6b0ceb700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac0603b0 con 0x7fc6ac0f55e0 2021-11-20T05:34:04.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.147+0000 7fc6b0ceb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac0604f0 con 0x7fc6ac0f5ef0 2021-11-20T05:34:04.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.147+0000 7fc6aa59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc6ac0f55e0 0x7fc6ac10c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:04.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.147+0000 7fc6aa59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc6ac0f55e0 0x7fc6ac10c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40472/0 (socket says 172.21.15.58:40472) 2021-11-20T05:34:04.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.147+0000 7fc6aa59c700 1 -- 172.21.15.58:0/777637736 learned_addr learned my addr 172.21.15.58:0/777637736 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:04.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.147+0000 7fc6a9d9b700 1 --2- 172.21.15.58:0/777637736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6ac0f5ef0 0x7fc6ac10e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:04.152 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.147+0000 7fc6aa59c700 1 -- 172.21.15.58:0/777637736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6ac0f5ef0 msgr2=0x7fc6ac10e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:04.152 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.147+0000 7fc6aa59c700 1 --2- 172.21.15.58:0/777637736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6ac0f5ef0 0x7fc6ac10e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:04.152 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.147+0000 7fc6aa59c700 1 -- 172.21.15.58:0/777637736 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c005040 con 0x7fc6ac0f55e0 2021-11-20T05:34:04.152 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.147+0000 7fc6aa59c700 1 --2- 172.21.15.58:0/777637736 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc6ac0f55e0 0x7fc6ac10c0b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fc69c009830 tx=0x7fc69c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:04.153 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.148+0000 7fc69affd700 1 -- 172.21.15.58:0/777637736 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc69c018070 con 0x7fc6ac0f55e0 2021-11-20T05:34:04.153 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.148+0000 7fc6b0ceb700 1 -- 172.21.15.58:0/777637736 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ac10eaa0 con 0x7fc6ac0f55e0 2021-11-20T05:34:04.153 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.148+0000 7fc6b0ceb700 1 -- 172.21.15.58:0/777637736 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ac10eec0 con 0x7fc6ac0f55e0 2021-11-20T05:34:04.153 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.148+0000 7fc69affd700 1 -- 172.21.15.58:0/777637736 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc69c013410 con 0x7fc6ac0f55e0 2021-11-20T05:34:04.153 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.148+0000 7fc69affd700 1 -- 172.21.15.58:0/777637736 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc69c01c620 con 0x7fc6ac0f55e0 2021-11-20T05:34:04.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.149+0000 7fc6b0ceb700 1 -- 172.21.15.58:0/777637736 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6ac045ad0 con 0x7fc6ac0f55e0 2021-11-20T05:34:04.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.149+0000 7fc69affd700 1 -- 172.21.15.58:0/777637736 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc69c01c890 con 0x7fc6ac0f55e0 2021-11-20T05:34:04.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.149+0000 7fc69affd700 1 --2- 172.21.15.58:0/777637736 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc69405c2b0 0x7fc69405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:04.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.149+0000 7fc69affd700 1 -- 172.21.15.58:0/777637736 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fc69c077990 con 0x7fc6ac0f55e0 2021-11-20T05:34:04.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.149+0000 7fc6a9d9b700 1 --2- 172.21.15.58:0/777637736 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc69405c2b0 0x7fc69405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:04.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.150+0000 7fc6a9d9b700 1 --2- 172.21.15.58:0/777637736 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc69405c2b0 0x7fc69405e770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fc6a000b790 tx=0x7fc6a0008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:04.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.153+0000 7fc69affd700 1 -- 172.21.15.58:0/777637736 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc69c04a9b0 con 0x7fc6ac0f55e0 2021-11-20T05:34:04.341 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:04.341+0000 7fc6b0ceb700 1 -- 172.21.15.58:0/777637736 --> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6ac0fdf30 con 0x7fc69405c2b0 2021-11-20T05:34:04.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:04 smithi058 conmon[33537]: audit 2021-11-20T05:34:02.444740+0000 mgr.smithi058.zagdlk (mgr.14164) 184 : audit [DBG] from='client.14416 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:34:04.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:04 smithi187 conmon[36358]: audit 2021-11-20T05:34:02.444740+0000 mgr.smithi058.zagdlk (mgr.14164) 184 : audit [DBG] from='client.14416 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:34:05.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:05 smithi187 conmon[36358]: cluster 2021-11-20T05:34:04.208305+0000 mgr.smithi058.zagdlk (mgr.14164) 185 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:05.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:05 smithi187 conmon[36358]: audit 2021-11-20T05:34:05.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:05 smithi187 conmon[36358]: 2021-11-20T05:34:04.343644+0000 mgr.smithi058.zagdlk (mgr.14164) 186 : audit [DBG] from='client.14420 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:34:05.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:05 smithi187 conmon[36358]: audit 2021-11-20T05:34:04.344133+0000 mon.smithi058 (mon.0) 445 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2021-11-20T05:34:05.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:05 smithi058 conmon[33537]: cluster 2021-11-20T05:34:04.208305+0000 2021-11-20T05:34:05.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:05 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 185 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:05.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:05 smithi058 conmon[33537]: audit 2021-11-20T05:34: 2021-11-20T05:34:05.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:05 smithi058 conmon[33537]: 04.343644+0000 mgr.smithi058.zagdlk (mgr.14164) 186 : audit [DBG] from='client.14420 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:34:05.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:05 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:34:05.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:05 smithi058 conmon[33537]: -20T05:34:04.344133+0000 mon.smithi058 (mon. 2021-11-20T05:34:05.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:05 smithi058 conmon[33537]: 0) 445 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2021-11-20T05:34:06.333 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:06.333+0000 7fc69affd700 1 -- 172.21.15.58:0/777637736 <== mgr.14164 v2:172.21.15.58:6800/2097301011 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc6ac0fdf30 con 0x7fc69405c2b0 2021-11-20T05:34:06.336 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:06.335+0000 7fc6b0ceb700 1 -- 172.21.15.58:0/777637736 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc69405c2b0 msgr2=0x7fc69405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:06.336 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:06.335+0000 7fc6b0ceb700 1 --2- 172.21.15.58:0/777637736 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc69405c2b0 0x7fc69405e770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fc6a000b790 tx=0x7fc6a0008040).stop 2021-11-20T05:34:06.336 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:06.335+0000 7fc6b0ceb700 1 -- 172.21.15.58:0/777637736 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc6ac0f55e0 msgr2=0x7fc6ac10c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:06.337 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:06.335+0000 7fc6b0ceb700 1 --2- 172.21.15.58:0/777637736 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc6ac0f55e0 0x7fc6ac10c0b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fc69c009830 tx=0x7fc69c009b30).stop 2021-11-20T05:34:06.337 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:06.336+0000 7fc6b0ceb700 1 -- 172.21.15.58:0/777637736 shutdown_connections 2021-11-20T05:34:06.337 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:06.336+0000 7fc6b0ceb700 1 --2- 172.21.15.58:0/777637736 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc69405c2b0 0x7fc69405e770 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:06.337 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:06.336+0000 7fc6b0ceb700 1 --2- 172.21.15.58:0/777637736 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc6ac0f55e0 0x7fc6ac10c0b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:06.338 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:06.336+0000 7fc6b0ceb700 1 --2- 172.21.15.58:0/777637736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc6ac0f5ef0 0x7fc6ac10e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:06.338 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:06.336+0000 7fc6b0ceb700 1 -- 172.21.15.58:0/777637736 >> 172.21.15.58:0/777637736 conn(0x7fc6ac0f0de0 msgr2=0x7fc6ac0fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:06.338 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:06.336+0000 7fc6b0ceb700 1 -- 172.21.15.58:0/777637736 shutdown_connections 2021-11-20T05:34:06.338 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:06.336+0000 7fc6b0ceb700 1 -- 172.21.15.58:0/777637736 wait complete. 2021-11-20T05:34:06.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:06 smithi187 conmon[36358]: audit 2021-11-20T05:34:05.291721+0000 mon.smithi058 (mon 2021-11-20T05:34:06.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:06 smithi187 conmon[36358]: .0) 446 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2021-11-20T05:34:06.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:06 smithi187 conmon[36358]: cluster 2021-11-20T05:34:05.291821+0000 mon.smithi058 (mon.0) 447 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-11-20T05:34:06.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:06 smithi187 conmon[36358]: audit 2021-11-20T05:34:05.294874+0000 2021-11-20T05:34:06.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:06 smithi187 conmon[36358]: mon.smithi058 (mon.0) 448 : audit 2021-11-20T05:34:06.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:06 smithi187 conmon[36358]: [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2021-11-20T05:34:06.707 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:06 smithi058 conmon[33537]: audit 2021-11-20T05:34:05.291721+0000 mon.smithi058 (mon.0) 446 : audit 2021-11-20T05:34:06.708 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:06 smithi058 conmon[33537]: [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2021-11-20T05:34:06.710 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:06 smithi058 conmon[33537]: cluster 2021-11-20T05:34:05.291821+0000 mon.smithi058 (mon.0) 447 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-11-20T05:34:06.710 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:06 smithi058 conmon[33537]: audit 2021-11-20T05:34:05.294874+0000 mon.smithi058 (mon. 2021-11-20T05:34:06.710 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:06 smithi058 conmon[33537]: 0) 448 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2021-11-20T05:34:06.711 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:06 smithi058 conmon[33537]: debug 2021-11-20T05:34:06.305+0000 7ff1a5efb700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T05:34:06.736 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs dump' 2021-11-20T05:34:07.510 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: cluster 2021-11-20T05:34:06.208665+0000 mgr.smithi058.zagdlk (mgr.14164) 187 : cluster [DBG] pgmap v150: 33 pgs: 5 creating+peering, 1 active+clean, 27 unknown; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:07.511 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: audit 2021-11-20T05:34:06.298162+0000 mon.smithi058 (mon.0) 449 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2021-11-20T05:34:07.511 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: cluster 2021-11-20T05:34:06.298323+0000 mon.smithi058 (mon.0) 450 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2021-11-20T05:34:07.511 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: audit 2021-11-20T05:34:06.305536+0000 mon.smithi058 (mon.0) 451 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2021-11-20T05:34:07.511 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: cluster 2021-11-20T05:34:06.306368+0000 mon.smithi058 (mon.0) 452 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T05:34:07.512 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: cluster 2021-11-20T05:34:06.306409+0000 mon.smithi058 (mon.0) 453 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2021-11-20T05:34:07.512 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: cluster 2021-11-20T05:34:06.321350+0000 mon.smithi058 (mon.0) 454 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2021-11-20T05:34:07.512 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: audit 2021-11-20T05:34:06.321678+0000 mon.smithi058 2021-11-20T05:34:07.513 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: (mon.0) 455 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2021-11-20T05:34:07.513 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: cluster 2021-11-20T05:34:06.321763+0000 mon.smithi058 (mon.0) 456 : cluster [DBG] fsmap cephfs:0 2021-11-20T05:34:07.513 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: cephadm 2021-11-20T05:34:06.324313+0000 mgr.smithi058.zagdlk (mgr.14164) 188 : cephadm [INF] Saving service mds.cephfs spec with placement count:4 2021-11-20T05:34:07.514 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: audit 2021-11-20T05:34:06.333108+0000 mon.smithi058 (mon.0) 457 : audit [INF] 2021-11-20T05:34:07.514 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:07.514 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: audit 2021-11-20T05:34:06. 2021-11-20T05:34:07.515 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: 334453+0000 mon.smithi058 (mon.0) 458 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:34:07.515 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: 2021-11-20T05:34:07.515 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: audit 2021-11-20T05:34:07.516 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: 2021-11-20T05:34:06.713240+0000 2021-11-20T05:34:07.516 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: mon.smithi058 (mon.0) 459 : audit [INF] 2021-11-20T05:34:07.516 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:07.516 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: audit 2021-11-20T05:34:07.517 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: 2021-11-20T05:34:06.719886+0000 2021-11-20T05:34:07.517 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: mon.smithi058 (mon.0) 460 : audit 2021-11-20T05:34:07.517 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:07.518 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: audit 2021-11-20T05: 2021-11-20T05:34:07.518 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: 34:06.720787+0000 mon.smithi058 (mon. 2021-11-20T05:34:07.518 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: 0) 461 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi058.rxapzd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T05:34:07.519 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:34:07.519 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: -20T05:34:06.724211+0000 2021-11-20T05:34:07.519 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: mon.smithi058 (mon.0) 462 : audit [INF] 2021-11-20T05:34:07.520 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi058.rxapzd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T05:34:07.521 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: audit 2021-11-20T05:34:07.521 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: 2021-11-20T05:34:06 2021-11-20T05:34:07.521 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: .725049+0000 mon.smithi058 (mon.0) 463 : 2021-11-20T05:34:07.522 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:07 smithi058 conmon[33537]: audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:34:07.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: cluster 2021-11-20T05:34:06.208665+0000 mgr.smithi058.zagdlk (mgr.14164) 187 : cluster [DBG] pgmap v150: 33 pgs: 5 creating+peering, 1 active+clean, 27 unknown; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:07.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: audit 2021-11-20T05:34:06.298162+0000 mon.smithi058 (mon.0) 449 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2021-11-20T05:34:07.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: cluster 2021-11-20T05:34:06.298323+0000 mon.smithi058 (mon.0) 450 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2021-11-20T05:34:07.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: audit 2021-11-20T05:34:06.305536+0000 mon.smithi058 (mon.0) 2021-11-20T05:34:07.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: 451 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2021-11-20T05:34:07.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: cluster 2021-11-20 2021-11-20T05:34:07.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: T05:34:06.306368+0000 mon.smithi058 (mon. 2021-11-20T05:34:07.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: 0) 452 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T05:34:07.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: cluster 2021-11-20T05:34:06.306409 2021-11-20T05:34:07.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 453 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2021-11-20T05:34:07.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: cluster 2021-11 2021-11-20T05:34:07.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: -20T05:34:06.321350+0000 mon.smithi058 (mon 2021-11-20T05:34:07.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: .0) 454 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2021-11-20T05:34:07.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: audit 2021-11-20 2021-11-20T05:34:07.702 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: T05:34:06.321678+0000 mon.smithi058 (mon. 2021-11-20T05:34:07.702 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: 0) 455 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2021-11-20T05:34:07.702 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: cluster 2021-11 2021-11-20T05:34:07.703 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: -20T05:34:06.321763+0000 mon.smithi058 ( 2021-11-20T05:34:07.703 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: mon.0) 456 : cluster [DBG] fsmap cephfs:0 2021-11-20T05:34:07.703 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: cephadm 2021 2021-11-20T05:34:07.703 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: -11-20T05:34:06.324313+0000 mgr.smithi058.zagdlk 2021-11-20T05:34:07.704 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: (mgr.14164) 188 : cephadm [INF] Saving service mds.cephfs spec with placement count:4 2021-11-20T05:34:07.704 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: audit 2021-11 2021-11-20T05:34:07.704 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: -20T05:34:06.333108+0000 mon.smithi058 ( 2021-11-20T05:34:07.704 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: mon.0) 457 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:07.705 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: audit 2021-11 2021-11-20T05:34:07.705 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: -20T05:34:06.334453+0000 mon.smithi058 2021-11-20T05:34:07.705 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: (mon.0) 458 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:34:07.705 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: audit 2021 2021-11-20T05:34:07.706 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: -11-20T05:34:06.713240 2021-11-20T05:34:07.706 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 459 : audit [INF] 2021-11-20T05:34:07.706 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:07.706 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: audit 2021-11-20T05:34:06 2021-11-20T05:34:07.707 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: .719886+0000 mon.smithi058 (mon.0) 460 : audit 2021-11-20T05:34:07.707 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:07.707 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: audit 2021-11 2021-11-20T05:34:07.707 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: -20T05:34:06.720787+0000 mon.smithi058 2021-11-20T05:34:07.708 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: (mon.0) 461 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi058.rxapzd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T05:34:07.708 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: audit 2021-11 2021-11-20T05:34:07.708 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: -20T05:34:06.724211+0000 mon.smithi058 ( 2021-11-20T05:34:07.708 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: mon.0) 462 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi058.rxapzd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T05:34:07.709 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: audit 2021-11-20T 2021-11-20T05:34:07.709 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: 05:34:06.725049+0000 mon.smithi058 (mon.0) 463 : audit 2021-11-20T05:34:07.709 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:07 smithi187 conmon[36358]: [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:34:08.593 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:08 smithi058 conmon[33537]: cephadm 2021-11-20T05:34:06. 2021-11-20T05:34:08.593 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:08 smithi058 conmon[33537]: 725850+0000 mgr.smithi058.zagdlk (mgr.14164) 189 : cephadm [INF] Deploying daemon mds.cephfs.smithi058.rxapzd on smithi058 2021-11-20T05:34:08.593 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:08 smithi058 conmon[33537]: cluster 2021-11-20T05:34:07.321321+0000 mon.smithi058 (mon.0) 464 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2021-11-20T05:34:08.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:08 smithi187 conmon[36358]: cephadm 2021-11-20T05:34:06.725850+0000 mgr.smithi058.zagdlk (mgr.14164) 189 : cephadm [INF] 2021-11-20T05:34:08.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:08 smithi187 conmon[36358]: Deploying daemon mds.cephfs.smithi058.rxapzd on smithi058 2021-11-20T05:34:08.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:08 smithi187 conmon[36358]: cluster 2021-11-20T05:34:07.321321+0000 mon.smithi058 (mon.0) 464 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2021-11-20T05:34:09.011 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.009+0000 7f544e8db700 1 -- 172.21.15.58:0/65139197 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f54480f55c0 msgr2=0x7f54480f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:09.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.009+0000 7f544e8db700 1 --2- 172.21.15.58:0/65139197 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f54480f55c0 0x7f54480f59e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f5438004660 tx=0x7f5438009b30).stop 2021-11-20T05:34:09.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.009+0000 7f544e8db700 1 -- 172.21.15.58:0/65139197 shutdown_connections 2021-11-20T05:34:09.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.009+0000 7f544e8db700 1 --2- 172.21.15.58:0/65139197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54480f6860 0x7f54480f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:09.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.009+0000 7f544e8db700 1 --2- 172.21.15.58:0/65139197 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f54480f55c0 0x7f54480f59e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:09.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.009+0000 7f544e8db700 1 -- 172.21.15.58:0/65139197 >> 172.21.15.58:0/65139197 conn(0x7f54480f0d40 msgr2=0x7f54480f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:09.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.010+0000 7f544e8db700 1 -- 172.21.15.58:0/65139197 shutdown_connections 2021-11-20T05:34:09.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.010+0000 7f544e8db700 1 -- 172.21.15.58:0/65139197 wait complete. 2021-11-20T05:34:09.014 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.010+0000 7f544e8db700 1 Processor -- start 2021-11-20T05:34:09.014 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.010+0000 7f544e8db700 1 -- start start 2021-11-20T05:34:09.014 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.010+0000 7f544e8db700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f54480f55c0 0x7f54480f9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:09.014 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.010+0000 7f544e8db700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54480f6860 0x7f54480f83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:09.015 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.011+0000 7f544e8db700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54480f8920 con 0x7f54480f55c0 2021-11-20T05:34:09.015 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.011+0000 7f544e8db700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54480f8a60 con 0x7f54480f6860 2021-11-20T05:34:09.015 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.011+0000 7f5447fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f54480f55c0 0x7f54480f9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:09.015 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.011+0000 7f5447fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f54480f55c0 0x7f54480f9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40482/0 (socket says 172.21.15.58:40482) 2021-11-20T05:34:09.016 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.011+0000 7f5447fff700 1 -- 172.21.15.58:0/2391956533 learned_addr learned my addr 172.21.15.58:0/2391956533 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:09.016 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.011+0000 7f54477fe700 1 --2- 172.21.15.58:0/2391956533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54480f6860 0x7f54480f83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:09.016 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.011+0000 7f5447fff700 1 -- 172.21.15.58:0/2391956533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54480f6860 msgr2=0x7f54480f83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:09.016 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.011+0000 7f5447fff700 1 --2- 172.21.15.58:0/2391956533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54480f6860 0x7f54480f83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:09.017 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.011+0000 7f5447fff700 1 -- 172.21.15.58:0/2391956533 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5438005040 con 0x7f54480f55c0 2021-11-20T05:34:09.017 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.011+0000 7f5447fff700 1 --2- 172.21.15.58:0/2391956533 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f54480f55c0 0x7f54480f9f40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f5438000c00 tx=0x7f543800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:09.017 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.011+0000 7f5444ff9700 1 -- 172.21.15.58:0/2391956533 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5438019070 con 0x7f54480f55c0 2021-11-20T05:34:09.017 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.011+0000 7f5444ff9700 1 -- 172.21.15.58:0/2391956533 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5438007e70 con 0x7f54480f55c0 2021-11-20T05:34:09.018 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.011+0000 7f544e8db700 1 -- 172.21.15.58:0/2391956533 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54480f8ce0 con 0x7f54480f55c0 2021-11-20T05:34:09.018 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.012+0000 7f5444ff9700 1 -- 172.21.15.58:0/2391956533 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5438014c90 con 0x7f54480f55c0 2021-11-20T05:34:09.018 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.012+0000 7f544e8db700 1 -- 172.21.15.58:0/2391956533 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54480f9240 con 0x7f54480f55c0 2021-11-20T05:34:09.019 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.013+0000 7f544e8db700 1 -- 172.21.15.58:0/2391956533 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5448045ad0 con 0x7f54480f55c0 2021-11-20T05:34:09.020 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.014+0000 7f5444ff9700 1 -- 172.21.15.58:0/2391956533 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5438025070 con 0x7f54480f55c0 2021-11-20T05:34:09.020 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.014+0000 7f5444ff9700 1 --2- 172.21.15.58:0/2391956533 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f543005c2a0 0x7f543005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:09.020 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.015+0000 7f5444ff9700 1 -- 172.21.15.58:0/2391956533 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f54380771c0 con 0x7f54480f55c0 2021-11-20T05:34:09.020 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.015+0000 7f54477fe700 1 --2- 172.21.15.58:0/2391956533 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f543005c2a0 0x7f543005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:09.021 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.015+0000 7f54477fe700 1 --2- 172.21.15.58:0/2391956533 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f543005c2a0 0x7f543005e760 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f543c009fd0 tx=0x7f543c00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:09.021 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.018+0000 7f5444ff9700 1 -- 172.21.15.58:0/2391956533 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f543804de60 con 0x7f54480f55c0 2021-11-20T05:34:09.221 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.221+0000 7f544e8db700 1 -- 172.21.15.58:0/2391956533 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f544810f700 con 0x7f54480f55c0 2021-11-20T05:34:09.222 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.221+0000 7f5444ff9700 1 -- 172.21.15.58:0/2391956533 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1003 (secure 0 0 0) 0x7f5438023070 con 0x7f54480f55c0 2021-11-20T05:34:09.222 INFO:teuthology.orchestra.run.smithi058.stdout:e2 2021-11-20T05:34:09.223 INFO:teuthology.orchestra.run.smithi058.stdout:enable_multiple, ever_enabled_multiple: 1,1 2021-11-20T05:34:09.223 INFO:teuthology.orchestra.run.smithi058.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T05:34:09.223 INFO:teuthology.orchestra.run.smithi058.stdout:legacy client fscid: 1 2021-11-20T05:34:09.223 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:34:09.224 INFO:teuthology.orchestra.run.smithi058.stdout:Filesystem 'cephfs' (1) 2021-11-20T05:34:09.224 INFO:teuthology.orchestra.run.smithi058.stdout:fs_name cephfs 2021-11-20T05:34:09.224 INFO:teuthology.orchestra.run.smithi058.stdout:epoch 2 2021-11-20T05:34:09.224 INFO:teuthology.orchestra.run.smithi058.stdout:flags 12 2021-11-20T05:34:09.224 INFO:teuthology.orchestra.run.smithi058.stdout:created 2021-11-20T05:34:06.306214+0000 2021-11-20T05:34:09.225 INFO:teuthology.orchestra.run.smithi058.stdout:modified 2021-11-20T05:34:06.306282+0000 2021-11-20T05:34:09.225 INFO:teuthology.orchestra.run.smithi058.stdout:tableserver 0 2021-11-20T05:34:09.225 INFO:teuthology.orchestra.run.smithi058.stdout:root 0 2021-11-20T05:34:09.225 INFO:teuthology.orchestra.run.smithi058.stdout:session_timeout 60 2021-11-20T05:34:09.226 INFO:teuthology.orchestra.run.smithi058.stdout:session_autoclose 300 2021-11-20T05:34:09.226 INFO:teuthology.orchestra.run.smithi058.stdout:max_file_size 1099511627776 2021-11-20T05:34:09.226 INFO:teuthology.orchestra.run.smithi058.stdout:required_client_features {} 2021-11-20T05:34:09.226 INFO:teuthology.orchestra.run.smithi058.stdout:last_failure 0 2021-11-20T05:34:09.227 INFO:teuthology.orchestra.run.smithi058.stdout:last_failure_osd_epoch 0 2021-11-20T05:34:09.227 INFO:teuthology.orchestra.run.smithi058.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T05:34:09.227 INFO:teuthology.orchestra.run.smithi058.stdout:max_mds 1 2021-11-20T05:34:09.227 INFO:teuthology.orchestra.run.smithi058.stdout:in 2021-11-20T05:34:09.228 INFO:teuthology.orchestra.run.smithi058.stdout:up {} 2021-11-20T05:34:09.228 INFO:teuthology.orchestra.run.smithi058.stdout:failed 2021-11-20T05:34:09.228 INFO:teuthology.orchestra.run.smithi058.stdout:damaged 2021-11-20T05:34:09.228 INFO:teuthology.orchestra.run.smithi058.stdout:stopped 2021-11-20T05:34:09.229 INFO:teuthology.orchestra.run.smithi058.stdout:data_pools [3] 2021-11-20T05:34:09.229 INFO:teuthology.orchestra.run.smithi058.stdout:metadata_pool 2 2021-11-20T05:34:09.229 INFO:teuthology.orchestra.run.smithi058.stdout:inline_data disabled 2021-11-20T05:34:09.229 INFO:teuthology.orchestra.run.smithi058.stdout:balancer 2021-11-20T05:34:09.230 INFO:teuthology.orchestra.run.smithi058.stdout:standby_count_wanted 0 2021-11-20T05:34:09.230 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:34:09.230 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:34:09.231 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.223+0000 7f544e8db700 1 -- 172.21.15.58:0/2391956533 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f543005c2a0 msgr2=0x7f543005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:09.231 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.223+0000 7f544e8db700 1 --2- 172.21.15.58:0/2391956533 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f543005c2a0 0x7f543005e760 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f543c009fd0 tx=0x7f543c00b040).stop 2021-11-20T05:34:09.232 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.223+0000 7f544e8db700 1 -- 172.21.15.58:0/2391956533 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f54480f55c0 msgr2=0x7f54480f9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:09.232 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.223+0000 7f544e8db700 1 --2- 172.21.15.58:0/2391956533 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f54480f55c0 0x7f54480f9f40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f5438000c00 tx=0x7f543800b040).stop 2021-11-20T05:34:09.232 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.223+0000 7f544e8db700 1 -- 172.21.15.58:0/2391956533 shutdown_connections 2021-11-20T05:34:09.232 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.223+0000 7f544e8db700 1 --2- 172.21.15.58:0/2391956533 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f543005c2a0 0x7f543005e760 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:09.233 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.223+0000 7f544e8db700 1 --2- 172.21.15.58:0/2391956533 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f54480f55c0 0x7f54480f9f40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:09.233 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.223+0000 7f544e8db700 1 --2- 172.21.15.58:0/2391956533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f54480f6860 0x7f54480f83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:09.233 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.223+0000 7f544e8db700 1 -- 172.21.15.58:0/2391956533 >> 172.21.15.58:0/2391956533 conn(0x7f54480f0d40 msgr2=0x7f544805f720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:09.233 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.224+0000 7f544e8db700 1 -- 172.21.15.58:0/2391956533 shutdown_connections 2021-11-20T05:34:09.233 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:09.224+0000 7f544e8db700 1 -- 172.21.15.58:0/2391956533 wait complete. 2021-11-20T05:34:09.234 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 2 2021-11-20T05:34:09.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:09 smithi058 conmon[33537]: cluster 2021-11-20T05:34:08.209095 2021-11-20T05:34:09.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:09 smithi058 conmon[33537]: +0000 mgr.smithi058.zagdlk (mgr.14164) 190 : cluster [DBG] pgmap v154: 65 pgs: 11 creating+peering, 19 active+clean, 35 unknown; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:09.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:09 smithi058 conmon[33537]: cluster 2021-11-20T05:34:08.326027+0000 mon.smithi058 (mon 2021-11-20T05:34:09.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:09 smithi058 conmon[33537]: .0) 465 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2021-11-20T05:34:09.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:09 smithi058 conmon[33537]: audit 2021-11-20T05:34:09. 2021-11-20T05:34:09.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:09 smithi058 conmon[33537]: 222511+0000 mon.smithi058 (mon.0) 466 : audit [DBG] from='client.? 172.21.15.58:0/2391956533' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-20T05:34:09.612 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T05:34:09.626 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi058.front.sepia.ceph.com 2021-11-20T05:34:09.627 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs set cephfs max_mds 2' 2021-11-20T05:34:09.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:09 smithi187 conmon[36358]: cluster 2021-11-20T05:34:08.209095+0000 mgr.smithi058.zagdlk (mgr.14164) 190 : cluster [DBG] pgmap v154: 65 pgs: 11 creating+peering, 19 active+clean, 35 unknown; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:09.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:09 smithi187 conmon[36358]: cluster 2021-11-20T05:34:08 2021-11-20T05:34:09.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:09 smithi187 conmon[36358]: .326027+0000 mon.smithi058 (mon.0) 465 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2021-11-20T05:34:09.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:09 smithi187 conmon[36358]: audit 2021-11-20T05:34:09.222511+0000 mon.smithi058 (mon.0) 466 : audit [DBG] from='client.? 172.21.15.58:0/2391956533' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-20T05:34:10.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:10 smithi187 conmon[36358]: audit 2021-11-20T05:34:09.694890+0000 mon.smithi058 (mon.0) 467 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:10.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:10 smithi187 conmon[36358]: audit 2021-11-20T05:34:09.695740+0000 mon.smithi058 (mon.0) 468 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi187.wxduhu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T05:34:10.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:10 smithi187 conmon[36358]: audit 2021-11-20T05:34:09.698783+0000 mon.smithi058 (mon.0) 469 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi187.wxduhu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T05:34:10.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:10 smithi187 conmon[36358]: audit 2021-11-20T05:34:09.699572+0000 mon.smithi058 (mon.0) 470 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:34:10.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:10 smithi187 conmon[36358]: cephadm 2021-11-20T05:34:09.700253+0000 mgr.smithi058.zagdlk (mgr.14164) 191 : cephadm [INF] Deploying daemon mds.cephfs.smithi187.wxduhu on smithi187 2021-11-20T05:34:10.949 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:10 smithi187 conmon[36358]: cluster 2021-11-20T05:34:10.209478+0000 mgr.smithi058.zagdlk (mgr.14164) 192 : cluster [DBG] pgmap v156: 65 pgs: 11 creating+peering, 46 active+clean, 8 unknown; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:10.962 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:10 smithi058 conmon[33537]: audit 2021-11-20T05:34:09.694890+0000 2021-11-20T05:34:10.962 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:10 smithi058 conmon[33537]: mon.smithi058 (mon.0) 467 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:10.962 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:10 smithi058 conmon[33537]: audit 2021-11-20T05:34:09.695740+0000 mon.smithi058 (mon.0) 468 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi187.wxduhu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T05:34:10.962 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:10 smithi058 conmon[33537]: audit 2021-11-20T05:34:09.698783+0000 mon.smithi058 (mon.0) 469 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi187.wxduhu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T05:34:10.963 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:10 smithi058 conmon[33537]: audit 2021-11-20T05:34:09.699572+0000 mon.smithi058 (mon.0) 470 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:34:10.963 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:10 smithi058 conmon[33537]: cephadm 2021-11-20T05:34:09.700253+0000 mgr.smithi058.zagdlk (mgr.14164) 191 : cephadm [INF] Deploying daemon mds.cephfs.smithi187.wxduhu on smithi187 2021-11-20T05:34:10.963 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:10 smithi058 conmon[33537]: cluster 2021-11-20T05:34:10.209478+0000 mgr.smithi058.zagdlk (mgr.14164) 192 : cluster [DBG] pgmap v156: 65 pgs: 11 creating+peering, 46 active+clean, 8 unknown; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:11.394 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.393+0000 7f4da9cf3700 1 -- 172.21.15.58:0/2907725450 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4da40f8610 msgr2=0x7f4da40f4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:11.395 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.393+0000 7f4da9cf3700 1 --2- 172.21.15.58:0/2907725450 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4da40f8610 0x7f4da40f4820 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f4d98004660 tx=0x7f4d98009b30).stop 2021-11-20T05:34:11.395 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.393+0000 7f4da9cf3700 1 -- 172.21.15.58:0/2907725450 shutdown_connections 2021-11-20T05:34:11.395 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.393+0000 7f4da9cf3700 1 --2- 172.21.15.58:0/2907725450 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4da40f8610 0x7f4da40f4820 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:11.395 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.393+0000 7f4da9cf3700 1 --2- 172.21.15.58:0/2907725450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4da40f7d00 0x7f4da40f8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:34:11.396 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.393+0000 7f4da9cf3700 1 -- 172.21.15.58:0/2907725450 >> 172.21.15.58:0/2907725450 conn(0x7f4da40f0540 msgr2=0x7f4da40f2960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:11.396 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.393+0000 7f4da9cf3700 1 -- 172.21.15.58:0/2907725450 shutdown_connections 2021-11-20T05:34:11.396 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.393+0000 7f4da9cf3700 1 -- 172.21.15.58:0/2907725450 wait complete. 2021-11-20T05:34:11.396 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.393+0000 7f4da9cf3700 1 Processor -- start 2021-11-20T05:34:11.396 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.393+0000 7f4da9cf3700 1 -- start start 2021-11-20T05:34:11.397 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4da9cf3700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4da40f7d00 0x7f4da4106f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:11.397 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4da9cf3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4da40f8610 0x7f4da4107450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:11.397 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4da9cf3700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da4107a90 con 0x7f4da40f7d00 2021-11-20T05:34:11.397 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4da9cf3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da4108340 con 0x7f4da40f8610 2021-11-20T05:34:11.398 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4da37fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4da40f7d00 0x7f4da4106f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:11.398 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4da37fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4da40f7d00 0x7f4da4106f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40500/0 (socket says 172.21.15.58:40500) 2021-11-20T05:34:11.398 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4da37fe700 1 -- 172.21.15.58:0/2507977754 learned_addr learned my addr 172.21.15.58:0/2507977754 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:11.398 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4da2ffd700 1 --2- 172.21.15.58:0/2507977754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4da40f8610 0x7f4da4107450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:11.399 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4da37fe700 1 -- 172.21.15.58:0/2507977754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4da40f8610 msgr2=0x7f4da4107450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:11.399 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4da37fe700 1 --2- 172.21.15.58:0/2507977754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4da40f8610 0x7f4da4107450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:11.399 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4da37fe700 1 -- 172.21.15.58:0/2507977754 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d98005040 con 0x7f4da40f7d00 2021-11-20T05:34:11.399 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4da2ffd700 1 --2- 172.21.15.58:0/2507977754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4da40f8610 0x7f4da4107450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:34:11.399 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4da37fe700 1 --2- 172.21.15.58:0/2507977754 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4da40f7d00 0x7f4da4106f60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f4d9400b410 tx=0x7f4d94006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:11.400 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4d8bfff700 1 -- 172.21.15.58:0/2507977754 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d940125f0 con 0x7f4da40f7d00 2021-11-20T05:34:11.400 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4d8bfff700 1 -- 172.21.15.58:0/2507977754 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d94012c00 con 0x7f4da40f7d00 2021-11-20T05:34:11.400 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4d8bfff700 1 -- 172.21.15.58:0/2507977754 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d94002aa0 con 0x7f4da40f7d00 2021-11-20T05:34:11.400 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4da9cf3700 1 -- 172.21.15.58:0/2507977754 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da4108620 con 0x7f4da40f7d00 2021-11-20T05:34:11.401 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.394+0000 7f4da9cf3700 1 -- 172.21.15.58:0/2507977754 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da4108be0 con 0x7f4da40f7d00 2021-11-20T05:34:11.402 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.395+0000 7f4da9cf3700 1 -- 172.21.15.58:0/2507977754 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da40f41d0 con 0x7f4da40f7d00 2021-11-20T05:34:11.402 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.396+0000 7f4d8bfff700 1 -- 172.21.15.58:0/2507977754 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4d9400f030 con 0x7f4da40f7d00 2021-11-20T05:34:11.402 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.396+0000 7f4d8bfff700 1 --2- 172.21.15.58:0/2507977754 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d8c05c2b0 0x7f4d8c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:11.403 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.396+0000 7f4d8bfff700 1 -- 172.21.15.58:0/2507977754 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f4d94076380 con 0x7f4da40f7d00 2021-11-20T05:34:11.403 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.396+0000 7f4da2ffd700 1 --2- 172.21.15.58:0/2507977754 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d8c05c2b0 0x7f4d8c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:11.403 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.397+0000 7f4da2ffd700 1 --2- 172.21.15.58:0/2507977754 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d8c05c2b0 0x7f4d8c05e770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f4d98009d60 tx=0x7f4d98009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:11.403 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.400+0000 7f4d8bfff700 1 -- 172.21.15.58:0/2507977754 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d94049040 con 0x7f4da40f7d00 2021-11-20T05:34:11.600 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.600+0000 7f4da9cf3700 1 -- 172.21.15.58:0/2507977754 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f4da4045ad0 con 0x7f4da40f7d00 2021-11-20T05:34:11.703 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.703+0000 7f4d8bfff700 1 -- 172.21.15.58:0/2507977754 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v3) v1 ==== 105+0+0 (secure 0 0 0) 0x7f4d9404c660 con 0x7f4da40f7d00 2021-11-20T05:34:11.705 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.705+0000 7f4da9cf3700 1 -- 172.21.15.58:0/2507977754 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d8c05c2b0 msgr2=0x7f4d8c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:11.705 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.705+0000 7f4da9cf3700 1 --2- 172.21.15.58:0/2507977754 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d8c05c2b0 0x7f4d8c05e770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f4d98009d60 tx=0x7f4d98009b30).stop 2021-11-20T05:34:11.705 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.705+0000 7f4da9cf3700 1 -- 172.21.15.58:0/2507977754 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4da40f7d00 msgr2=0x7f4da4106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:11.706 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.705+0000 7f4da9cf3700 1 --2- 172.21.15.58:0/2507977754 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4da40f7d00 0x7f4da4106f60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f4d9400b410 tx=0x7f4d94006ef0).stop 2021-11-20T05:34:11.706 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.705+0000 7f4da9cf3700 1 -- 172.21.15.58:0/2507977754 shutdown_connections 2021-11-20T05:34:11.706 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.705+0000 7f4da9cf3700 1 --2- 172.21.15.58:0/2507977754 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d8c05c2b0 0x7f4d8c05e770 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:11.706 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.705+0000 7f4da9cf3700 1 --2- 172.21.15.58:0/2507977754 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4da40f7d00 0x7f4da4106f60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:11.707 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.705+0000 7f4da9cf3700 1 --2- 172.21.15.58:0/2507977754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4da40f8610 0x7f4da4107450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:11.707 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.705+0000 7f4da9cf3700 1 -- 172.21.15.58:0/2507977754 >> 172.21.15.58:0/2507977754 conn(0x7f4da40f0540 msgr2=0x7f4da40fb590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:11.707 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.705+0000 7f4da9cf3700 1 -- 172.21.15.58:0/2507977754 shutdown_connections 2021-11-20T05:34:11.707 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:11.705+0000 7f4da9cf3700 1 -- 172.21.15.58:0/2507977754 wait complete. 2021-11-20T05:34:11.835 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:11 smithi187 conmon[36358]: audit 2021-11-20T05:34:11.601595+0000 mon.smithi058 (mon.0) 471 : audit [INF] from='client.? 172.21.15.58:0/2507977754' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2021-11-20T05:34:11.835 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:11 smithi187 conmon[36358]: 2021-11-20T05:34:11.969 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:11 smithi058 conmon[33537]: audit 2021-11-20T05:34:11.601595+0000 mon.smithi058 (mon.0) 471 : audit [INF] from='client.? 172.21.15.58:0/2507977754' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2021-11-20T05:34:12.157 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T05:34:12.171 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi058.front.sepia.ceph.com 2021-11-20T05:34:12.171 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2021-11-20T05:34:12.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: audit 2021-11-20T05:34:11.704444 2021-11-20T05:34:12.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 472 : audit [INF] from='client.? 172.21.15.58:0/2507977754' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2021-11-20T05:34:12.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: cluster 2021-11-20T05:34:11.704530+0000 mon.smithi058 (mon.0) 473 : cluster [DBG] fsmap cephfs:0 2021-11-20T05:34:12.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: audit 2021-11-20T05:34:11.877119+0000 mon.smithi058 (mon.0) 474 2021-11-20T05:34:12.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:12.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:34:12.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: 11-20T05:34:11.877877+0000 mon.smithi058 (mon. 2021-11-20T05:34:12.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: 0) 475 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi058.nuxpgf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T05:34:12.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: audit 2021-11- 2021-11-20T05:34:12.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: 20T05:34:11.880400+0000 mon.smithi058 (mon.0) 476 2021-11-20T05:34:12.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi058.nuxpgf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T05:34:12.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: audit 2021-11-20T05:34 2021-11-20T05:34:12.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: :11.881028+0000 mon.smithi058 (mon.0) 2021-11-20T05:34:12.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: 477 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:34:12.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: cephadm 2021-11-20T05:34:11. 2021-11-20T05:34:12.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: 881585+0000 mgr.smithi058.zagdlk (mgr.14164) 193 : cephadm [INF] 2021-11-20T05:34:12.973 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: Deploying daemon mds.cephfs.smithi058.nuxpgf on smithi058 2021-11-20T05:34:12.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: cluster 2021- 2021-11-20T05:34:12.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: 11-20T05:34:12.209998+0000 mgr.smithi058.zagdlk (mgr.14164) 194 2021-11-20T05:34:12.974 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:12 smithi058 conmon[33537]: : cluster [DBG] pgmap v157: 65 pgs: 65 active+clean; 0 B data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:13.197 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:12 smithi187 conmon[36358]: audit 2021-11-20T05:34:11.704444+0000 mon.smithi058 (mon.0) 472 : audit [INF] from='client.? 172.21.15.58:0/2507977754' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2021-11-20T05:34:13.197 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:12 smithi187 conmon[36358]: cluster 2021-11-20T05:34:11.704530+0000 mon.smithi058 (mon.0) 473 : cluster [DBG] fsmap cephfs:0 2021-11-20T05:34:13.198 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:12 smithi187 conmon[36358]: audit 2021-11-20T05:34:11.877119+0000 mon.smithi058 (mon.0) 474 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:13.198 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:12 smithi187 conmon[36358]: 2021-11-20T05:34:13.198 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:12 smithi187 conmon[36358]: audit 2021-11-20T05:34:11.877877+0000 mon.smithi058 (mon.0) 475 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi058.nuxpgf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T05:34:13.198 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:12 smithi187 conmon[36358]: audit 2021-11-20T05:34:11.880400+0000 mon.smithi058 (mon.0 2021-11-20T05:34:13.199 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:12 smithi187 conmon[36358]: ) 476 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi058.nuxpgf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T05:34:13.199 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:12 smithi187 conmon[36358]: audit 2021-11-20T05:34:11.881028+0000 mon.smithi058 (mon.0) 2021-11-20T05:34:13.199 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:12 smithi187 conmon[36358]: 477 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:34:13.199 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:12 smithi187 conmon[36358]: cephadm 2021-11-20T05:34:11.881585+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:34:13.199 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:12 smithi187 conmon[36358]: mgr.14164) 193 : cephadm [INF] Deploying daemon mds.cephfs.smithi058.nuxpgf on smithi058 2021-11-20T05:34:13.200 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:12 smithi187 conmon[36358]: cluster 2021-11-20T05:34: 2021-11-20T05:34:13.200 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:12 smithi187 conmon[36358]: 12.209998+0000 mgr.smithi058.zagdlk (mgr.14164) 194 : cluster [DBG] pgmap v157: 65 pgs: 65 active+clean; 0 B data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:13.200 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:12 smithi187 conmon[36358]: 2021-11-20T05:34:14.138 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.136+0000 7f4eb2359700 1 -- 172.21.15.58:0/3549765835 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4eac0f55c0 msgr2=0x7f4eac0f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:14.138 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.136+0000 7f4eb2359700 1 --2- 172.21.15.58:0/3549765835 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4eac0f55c0 0x7f4eac0f59e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f4e9c004660 tx=0x7f4e9c009b30).stop 2021-11-20T05:34:14.139 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.137+0000 7f4eb2359700 1 -- 172.21.15.58:0/3549765835 shutdown_connections 2021-11-20T05:34:14.139 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.137+0000 7f4eb2359700 1 --2- 172.21.15.58:0/3549765835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4eac0f6860 0x7f4eac0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:14.139 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.137+0000 7f4eb2359700 1 --2- 172.21.15.58:0/3549765835 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4eac0f55c0 0x7f4eac0f59e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:14.139 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.137+0000 7f4eb2359700 1 -- 172.21.15.58:0/3549765835 >> 172.21.15.58:0/3549765835 conn(0x7f4eac0f0d40 msgr2=0x7f4eac0f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:14.139 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.137+0000 7f4eb2359700 1 -- 172.21.15.58:0/3549765835 shutdown_connections 2021-11-20T05:34:14.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.137+0000 7f4eb2359700 1 -- 172.21.15.58:0/3549765835 wait complete. 2021-11-20T05:34:14.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.137+0000 7f4eb2359700 1 Processor -- start 2021-11-20T05:34:14.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.137+0000 7f4eb2359700 1 -- start start 2021-11-20T05:34:14.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.137+0000 7f4eb2359700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4eac0f55c0 0x7f4eac10c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:14.141 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.137+0000 7f4eb2359700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4eac0f6860 0x7f4eac10e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:14.141 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.137+0000 7f4eb2359700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac0602d0 con 0x7f4eac0f55c0 2021-11-20T05:34:14.141 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.137+0000 7f4eb2359700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac060410 con 0x7f4eac0f6860 2021-11-20T05:34:14.141 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.138+0000 7f4eabfff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4eac0f55c0 0x7f4eac10c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:14.141 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.138+0000 7f4eabfff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4eac0f55c0 0x7f4eac10c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40522/0 (socket says 172.21.15.58:40522) 2021-11-20T05:34:14.142 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.138+0000 7f4eabfff700 1 -- 172.21.15.58:0/3875864447 learned_addr learned my addr 172.21.15.58:0/3875864447 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:14.142 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.138+0000 7f4eab7fe700 1 --2- 172.21.15.58:0/3875864447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4eac0f6860 0x7f4eac10e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:14.142 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.138+0000 7f4eabfff700 1 -- 172.21.15.58:0/3875864447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4eac0f6860 msgr2=0x7f4eac10e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:14.142 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.138+0000 7f4eabfff700 1 --2- 172.21.15.58:0/3875864447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4eac0f6860 0x7f4eac10e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:14.144 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.138+0000 7f4eabfff700 1 -- 172.21.15.58:0/3875864447 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9c005040 con 0x7f4eac0f55c0 2021-11-20T05:34:14.144 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.138+0000 7f4eab7fe700 1 --2- 172.21.15.58:0/3875864447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4eac0f6860 0x7f4eac10e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:34:14.144 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.138+0000 7f4eabfff700 1 --2- 172.21.15.58:0/3875864447 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4eac0f55c0 0x7f4eac10c0d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4e9c004e30 tx=0x7f4e9c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:14.145 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.138+0000 7f4ea8ff9700 1 -- 172.21.15.58:0/3875864447 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e9c018070 con 0x7f4eac0f55c0 2021-11-20T05:34:14.145 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.138+0000 7f4ea8ff9700 1 -- 172.21.15.58:0/3875864447 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e9c013410 con 0x7f4eac0f55c0 2021-11-20T05:34:14.145 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.138+0000 7f4ea8ff9700 1 -- 172.21.15.58:0/3875864447 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e9c01c3c0 con 0x7f4eac0f55c0 2021-11-20T05:34:14.145 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.138+0000 7f4eb2359700 1 -- 172.21.15.58:0/3875864447 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eac10eac0 con 0x7f4eac0f55c0 2021-11-20T05:34:14.145 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.138+0000 7f4eb2359700 1 -- 172.21.15.58:0/3875864447 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eac10ee90 con 0x7f4eac0f55c0 2021-11-20T05:34:14.146 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.139+0000 7f4eb2359700 1 -- 172.21.15.58:0/3875864447 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eac045ad0 con 0x7f4eac0f55c0 2021-11-20T05:34:14.146 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.140+0000 7f4ea8ff9700 1 -- 172.21.15.58:0/3875864447 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4e9c01d420 con 0x7f4eac0f55c0 2021-11-20T05:34:14.146 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.140+0000 7f4ea8ff9700 1 --2- 172.21.15.58:0/3875864447 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4e9405c2b0 0x7f4e9405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:14.146 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.140+0000 7f4ea8ff9700 1 -- 172.21.15.58:0/3875864447 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f4e9c077ad0 con 0x7f4eac0f55c0 2021-11-20T05:34:14.147 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.140+0000 7f4eab7fe700 1 --2- 172.21.15.58:0/3875864447 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4e9405c2b0 0x7f4e9405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:14.147 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.141+0000 7f4eab7fe700 1 --2- 172.21.15.58:0/3875864447 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4e9405c2b0 0x7f4e9405e770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f4ea0009740 tx=0x7f4ea0009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:14.147 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.144+0000 7f4ea8ff9700 1 -- 172.21.15.58:0/3875864447 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e9c04a790 con 0x7f4eac0f55c0 2021-11-20T05:34:14.355 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.355+0000 7f4eb2359700 1 -- 172.21.15.58:0/3875864447 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4eac027a00 con 0x7f4eac0f55c0 2021-11-20T05:34:14.889 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.889+0000 7f4ea8ff9700 1 -- 172.21.15.58:0/3875864447 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4e9c04ddb0 con 0x7f4eac0f55c0 2021-11-20T05:34:14.892 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.892+0000 7f4eb2359700 1 -- 172.21.15.58:0/3875864447 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4e9405c2b0 msgr2=0x7f4e9405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:14.892 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.892+0000 7f4eb2359700 1 --2- 172.21.15.58:0/3875864447 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4e9405c2b0 0x7f4e9405e770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f4ea0009740 tx=0x7f4ea0009be0).stop 2021-11-20T05:34:14.892 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.892+0000 7f4eb2359700 1 -- 172.21.15.58:0/3875864447 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4eac0f55c0 msgr2=0x7f4eac10c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:14.893 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.892+0000 7f4eb2359700 1 --2- 172.21.15.58:0/3875864447 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4eac0f55c0 0x7f4eac10c0d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4e9c004e30 tx=0x7f4e9c009b30).stop 2021-11-20T05:34:14.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.893+0000 7f4eb2359700 1 -- 172.21.15.58:0/3875864447 shutdown_connections 2021-11-20T05:34:14.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.893+0000 7f4eb2359700 1 --2- 172.21.15.58:0/3875864447 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4e9405c2b0 0x7f4e9405e770 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:14.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.893+0000 7f4eb2359700 1 --2- 172.21.15.58:0/3875864447 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4eac0f55c0 0x7f4eac10c0d0 secure :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0x7f4e9c004e30 tx=0x7f4e9c009b30).stop 2021-11-20T05:34:14.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.893+0000 7f4eb2359700 1 --2- 172.21.15.58:0/3875864447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4eac0f6860 0x7f4eac10e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:14.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.893+0000 7f4eb2359700 1 -- 172.21.15.58:0/3875864447 >> 172.21.15.58:0/3875864447 conn(0x7f4eac0f0d40 msgr2=0x7f4eac0fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:14.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.894+0000 7f4eb2359700 1 -- 172.21.15.58:0/3875864447 shutdown_connections 2021-11-20T05:34:14.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:14.894+0000 7f4eb2359700 1 -- 172.21.15.58:0/3875864447 wait complete. 2021-11-20T05:34:15.184 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:14 smithi187 conmon[36358]: cluster 2021-11-20T05:34:13.883451+0000 mon.smithi058 (mon.0) 2021-11-20T05:34:15.184 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:14 smithi187 conmon[36358]: 478 : cluster [DBG] mds.? [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] up:boot 2021-11-20T05:34:15.184 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:14 smithi187 conmon[36358]: cluster 2021-11-20T05:34:13.883511+0000 mon.smithi058 (mon.0) 479 : cluster [INF] daemon mds.cephfs.smithi058.rxapzd assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2021-11-20T05:34:15.184 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:14 smithi187 conmon[36358]: cluster 2021-11-20T05:34:13.883604+0000 mon.smithi058 (mon.0) 480 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T05:34:15.185 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:14 smithi187 conmon[36358]: cluster 2021-11-20T05:34:13.883812+0000 mon.smithi058 (mon.0) 481 : cluster [DBG] fsmap cephfs:0 1 up:standby 2021-11-20T05:34:15.185 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:14 smithi187 conmon[36358]: audit 2021-11-20T05:34:13.883923+0000 mon.smithi058 (mon.0) 482 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi058.rxapzd"}]: dispatch 2021-11-20T05:34:15.185 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:14 smithi187 conmon[36358]: cluster 2021-11-20T05:34:13.889834+0000 mon.smithi058 (mon.0) 483 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi058.rxapzd=up:creating} 2021-11-20T05:34:15.185 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:14 smithi187 conmon[36358]: cluster 2021-11-20T05:34:13.901308+0000 mon.smithi058 (mon.0) 484 : cluster [INF] daemon mds.cephfs.smithi058.rxapzd is now active in filesystem cephfs as rank 0 2021-11-20T05:34:15.186 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:14 smithi187 conmon[36358]: cluster 2021-11-20T05:34:14.210451+0000 mgr.smithi058.zagdlk (mgr.14164) 195 : cluster [DBG] pgmap v158: 65 pgs: 65 active+clean; 160 B data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:15.186 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:14 smithi187 conmon[36358]: audit 2021-11-20T05:34:14.356880+0000 mon.smithi058 (mon.0) 485 : audit [INF] from='client.? 172.21.15.58:0/3875864447' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2021-11-20T05:34:15.186 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:14 smithi187 conmon[36358]: audit 2021-11-20T05:34:14.369749+0000 mon.smithi058 (mon.0) 486 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:15.186 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:14 smithi187 conmon[36358]: audit 2021-11-20T05:34:14.370573+0000 mon.smithi058 (mon.0) 487 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi187.fikisa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T05:34:15.187 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:14 smithi187 conmon[36358]: audit 2021-11-20T05:34:14.373502+0000 mon.smithi058 (mon.0) 488 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi187.fikisa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T05:34:15.187 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:14 smithi187 conmon[36358]: audit 2021-11-20T05:34:14.374180+0000 mon.smithi058 (mon.0) 489 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:34:15.187 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:14 smithi187 conmon[36358]: cephadm 2021-11-20T05:34:14.374743+0000 mgr.smithi058.zagdlk (mgr.14164) 196 : cephadm [INF] Deploying daemon mds.cephfs.smithi187.fikisa on smithi187 2021-11-20T05:34:15.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: cluster 2021-11-20T05:34:13.883451 2021-11-20T05:34:15.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 478 : cluster [DBG] mds.? [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] up:boot 2021-11-20T05:34:15.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: cluster 2021-11-20T05:34:13.883511+0000 mon.smithi058 (mon.0) 479 : cluster [INF] daemon mds.cephfs.smithi058.rxapzd assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2021-11-20T05:34:15.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: cluster 2021-11-20T05:34:13.883604+0000 mon.smithi058 (mon.0) 480 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T05:34:15.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: cluster 2021-11-20T05:34:13.883812+0000 mon.smithi058 (mon.0) 481 : cluster [DBG] fsmap cephfs:0 1 up:standby 2021-11-20T05:34:15.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: audit 2021-11-20T05 2021-11-20T05:34:15.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: :34:13.883923+0000 mon.smithi058 (mon.0) 482 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi058.rxapzd"}]: dispatch 2021-11-20T05:34:15.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: cluster 2021-11-20T05:34:13.889834+0000 mon.smithi058 (mon.0 2021-11-20T05:34:15.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: ) 483 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi058.rxapzd=up:creating} 2021-11-20T05:34:15.223 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: cluster 2021-11-20T05:34:15.223 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: 2021-11-20T05:34:13.901308+0000 mon.smithi058 (mon.0) 484 : cluster [INF] 2021-11-20T05:34:15.223 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: daemon mds.cephfs.smithi058.rxapzd is now active in filesystem cephfs as rank 0 2021-11-20T05:34:15.223 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: cluster 2021-11-20T 2021-11-20T05:34:15.223 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: 05:34:14.210451+0000 mgr.smithi058.zagdlk (mgr.14164) 195 : cluster [DBG] pgmap v158: 65 pgs: 65 active+clean; 160 B data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:15.224 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: audit 2021-11-20T05:34:14.356880+0000 2021-11-20T05:34:15.224 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: mon.smithi058 (mon.0) 485 : audit [INF] from='client.? 172.21.15.58:0/3875864447' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2021-11-20T05:34:15.225 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: audit 2021-11-20T05:34:14.369749+0000 mon.smithi058 (mon. 2021-11-20T05:34:15.225 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: 0) 486 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:15.225 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: audit 2021-11-20T05:34:15.225 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: 2021-11-20T05:34:14.370573+0000 mon.smithi058 (mon.0) 487 : audit 2021-11-20T05:34:15.226 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi187.fikisa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T05:34:15.226 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:34:15.226 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: -20T05:34:14.373502+0000 mon.smithi058 (mon.0) 488 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi187.fikisa", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T05:34:15.226 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: audit 2021-11-20T05:34:14. 2021-11-20T05:34:15.227 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: 374180+0000 mon.smithi058 (mon.0) 489 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:34:15.227 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: cephadm 2021-11-20T05:34:14 2021-11-20T05:34:15.227 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: .374743+0000 mgr.smithi058.zagdlk (mgr.14164) 196 : cephadm [INF] 2021-11-20T05:34:15.227 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:14 smithi058 conmon[33537]: Deploying daemon mds.cephfs.smithi187.fikisa on smithi187 2021-11-20T05:34:15.245 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T05:34:15.258 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi058.front.sepia.ceph.com 2021-11-20T05:34:15.258 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs dump' 2021-11-20T05:34:15.969 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:15 smithi058 conmon[33537]: cluster 2021-11-20T05:34:14.890340+0000 mon.smithi058 (mon.0) 490 : cluster [DBG] mds.? [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] up:active 2021-11-20T05:34:15.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:15 smithi058 conmon[33537]: cluster 2021-11-20T05:34:14.890410+0000 mon.smithi058 (mon.0) 491 : cluster [DBG] mds.? [v2:172.21.15.58:6828/4151830882,v1:172.21.15.58:6829/4151830882] up:boot 2021-11-20T05:34:15.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:15 smithi058 conmon[33537]: audit 2021-11-20T05:34:14.890510+0000 mon.smithi058 (mon.0) 492 : audit [INF] from='client.? 172.21.15.58:0/3875864447' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2021-11-20T05:34:15.970 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:15 smithi058 conmon[33537]: cluster 2021-11-20T05:34:14.890610+0000 mon.smithi058 (mon.0) 493 : cluster [INF] daemon mds.cephfs.smithi058.nuxpgf assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2021-11-20T05:34:15.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:15 smithi058 conmon[33537]: cluster 2021-11-20T05:34:14.890720+0000 mon.smithi058 (mon.0) 494 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2021-11-20T05:34:15.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:15 smithi058 conmon[33537]: cluster 2021-11-20T05:34:14.890746+0000 mon.smithi058 (mon.0) 495 : cluster 2021-11-20T05:34:15.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:15 smithi058 conmon[33537]: [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2021-11-20T05:34:15.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:15 smithi058 conmon[33537]: cluster 2021-11-20T05:34:14.890999+0000 mon.smithi058 (mon.0) 496 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi058.rxapzd=up:active} 1 up:standby 2021-11-20T05:34:15.971 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:15 smithi058 conmon[33537]: audit 2021-11-20T05:34:14.891138+0000 mon.smithi058 (mon.0) 497 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi058.nuxpgf"}]: dispatch 2021-11-20T05:34:15.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:15 smithi058 conmon[33537]: cluster 2021-11-20T05:34:14.896338+0000 mon.smithi058 (mon.0) 498 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi058.rxapzd=up:active,1=cephfs.smithi058.nuxpgf=up:creating} 2021-11-20T05:34:15.972 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:15 smithi058 conmon[33537]: cluster 2021-11-20T05:34:14.911879+0000 mon.smithi058 (mon.0) 499 : cluster [INF] daemon mds.cephfs.smithi058.nuxpgf is now active in filesystem cephfs as rank 1 2021-11-20T05:34:16.087 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:15 smithi187 conmon[36358]: cluster 2021-11-20T05:34:14.890340+0000 mon.smithi058 2021-11-20T05:34:16.087 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:15 smithi187 conmon[36358]: (mon.0) 490 : cluster [DBG] mds.? [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] up:active 2021-11-20T05:34:16.088 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:15 smithi187 conmon[36358]: cluster 2021-11-20T05:34:14.890410+0000 mon.smithi058 (mon.0) 491 : cluster [DBG] mds.? [v2:172.21.15.58:6828/4151830882,v1:172.21.15.58:6829/4151830882] up:boot 2021-11-20T05:34:16.088 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:15 smithi187 conmon[36358]: audit 2021-11-20T05:34:14.890510+0000 mon.smithi058 (mon.0) 492 : audit [INF] from='client.? 172.21.15.58:0/3875864447' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2021-11-20T05:34:16.088 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:15 smithi187 conmon[36358]: cluster 2021-11-20T05:34:14.890610+0000 mon.smithi058 (mon.0) 493 : cluster [INF] daemon mds.cephfs.smithi058.nuxpgf assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2021-11-20T05:34:16.088 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:15 smithi187 conmon[36358]: cluster 2021-11-20T05:34:14.890720+0000 mon.smithi058 (mon.0) 494 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2021-11-20T05:34:16.089 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:15 smithi187 conmon[36358]: cluster 2021-11-20T05:34:14.890746+0000 mon.smithi058 (mon.0) 495 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2021-11-20T05:34:16.089 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:15 smithi187 conmon[36358]: cluster 2021-11-20T05:34:14.890999+0000 mon.smithi058 (mon.0) 496 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi058.rxapzd=up:active} 1 up:standby 2021-11-20T05:34:16.089 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:15 smithi187 conmon[36358]: audit 2021-11-20T05:34:14.891138+0000 mon.smithi058 (mon.0) 497 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi058.nuxpgf"}]: dispatch 2021-11-20T05:34:16.089 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:15 smithi187 conmon[36358]: cluster 2021-11-20T05:34:14.896338+0000 mon.smithi058 (mon.0) 498 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi058.rxapzd=up:active,1=cephfs.smithi058.nuxpgf=up:creating} 2021-11-20T05:34:16.089 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:15 smithi187 conmon[36358]: cluster 2021-11-20T05:34:14.911879+0000 mon.smithi058 (mon.0) 499 : cluster [INF] daemon mds.cephfs.smithi058.nuxpgf is now active in filesystem cephfs as rank 1 2021-11-20T05:34:16.856 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.855+0000 7f84d8ead700 1 -- 172.21.15.58:0/515470220 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f84d40f55c0 msgr2=0x7f84d40f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:16.857 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.855+0000 7f84d8ead700 1 --2- 172.21.15.58:0/515470220 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f84d40f55c0 0x7f84d40f59e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f84c4004660 tx=0x7f84c4009b30).stop 2021-11-20T05:34:16.857 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.855+0000 7f84d8ead700 1 -- 172.21.15.58:0/515470220 shutdown_connections 2021-11-20T05:34:16.857 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.855+0000 7f84d8ead700 1 --2- 172.21.15.58:0/515470220 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f84d40f6860 0x7f84d40f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:16.857 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.855+0000 7f84d8ead700 1 --2- 172.21.15.58:0/515470220 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f84d40f55c0 0x7f84d40f59e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:16.858 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.855+0000 7f84d8ead700 1 -- 172.21.15.58:0/515470220 >> 172.21.15.58:0/515470220 conn(0x7f84d40f0d40 msgr2=0x7f84d40f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:16.858 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.855+0000 7f84d8ead700 1 -- 172.21.15.58:0/515470220 shutdown_connections 2021-11-20T05:34:16.858 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.855+0000 7f84d8ead700 1 -- 172.21.15.58:0/515470220 wait complete. 2021-11-20T05:34:16.858 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.856+0000 7f84d8ead700 1 Processor -- start 2021-11-20T05:34:16.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.856+0000 7f84d8ead700 1 -- start start 2021-11-20T05:34:16.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.856+0000 7f84d8ead700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f84d40f55c0 0x7f84d40f9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:16.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.856+0000 7f84d8ead700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f84d40f6860 0x7f84d40f83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:16.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.856+0000 7f84d8ead700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d40f8920 con 0x7f84d40f55c0 2021-11-20T05:34:16.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.856+0000 7f84d8ead700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d40f8a60 con 0x7f84d40f6860 2021-11-20T05:34:16.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.856+0000 7f84d259c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f84d40f55c0 0x7f84d40f9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:16.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.856+0000 7f84d259c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f84d40f55c0 0x7f84d40f9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40556/0 (socket says 172.21.15.58:40556) 2021-11-20T05:34:16.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.856+0000 7f84d259c700 1 -- 172.21.15.58:0/2954523915 learned_addr learned my addr 172.21.15.58:0/2954523915 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:16.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.856+0000 7f84d259c700 1 -- 172.21.15.58:0/2954523915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f84d40f6860 msgr2=0x7f84d40f83c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-11-20T05:34:16.861 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.856+0000 7f84d259c700 1 --2- 172.21.15.58:0/2954523915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f84d40f6860 0x7f84d40f83c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:34:16.861 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.856+0000 7f84d259c700 1 -- 172.21.15.58:0/2954523915 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c4005040 con 0x7f84d40f55c0 2021-11-20T05:34:16.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.857+0000 7f84d259c700 1 --2- 172.21.15.58:0/2954523915 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f84d40f55c0 0x7f84d40f9f40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f84c4004e30 tx=0x7f84c4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:16.864 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.857+0000 7f84c2ffd700 1 -- 172.21.15.58:0/2954523915 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84c4018070 con 0x7f84d40f55c0 2021-11-20T05:34:16.864 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.857+0000 7f84c2ffd700 1 -- 172.21.15.58:0/2954523915 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84c4013410 con 0x7f84d40f55c0 2021-11-20T05:34:16.865 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.857+0000 7f84c2ffd700 1 -- 172.21.15.58:0/2954523915 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84c401c3c0 con 0x7f84d40f55c0 2021-11-20T05:34:16.865 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.857+0000 7f84d8ead700 1 -- 172.21.15.58:0/2954523915 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d40f8ce0 con 0x7f84d40f55c0 2021-11-20T05:34:16.865 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.857+0000 7f84d8ead700 1 -- 172.21.15.58:0/2954523915 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d40f9240 con 0x7f84d40f55c0 2021-11-20T05:34:16.865 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.858+0000 7f84d8ead700 1 -- 172.21.15.58:0/2954523915 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d4045ad0 con 0x7f84d40f55c0 2021-11-20T05:34:16.866 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.860+0000 7f84c2ffd700 1 -- 172.21.15.58:0/2954523915 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f84c401d420 con 0x7f84d40f55c0 2021-11-20T05:34:16.866 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.860+0000 7f84c2ffd700 1 --2- 172.21.15.58:0/2954523915 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f84bc05c2b0 0x7f84bc05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:16.866 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.860+0000 7f84c2ffd700 1 -- 172.21.15.58:0/2954523915 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f84c4077d60 con 0x7f84d40f55c0 2021-11-20T05:34:16.866 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.860+0000 7f84d1d9b700 1 --2- 172.21.15.58:0/2954523915 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f84bc05c2b0 0x7f84bc05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:16.866 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.860+0000 7f84d1d9b700 1 --2- 172.21.15.58:0/2954523915 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f84bc05c2b0 0x7f84bc05e770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f84c8004ad0 tx=0x7f84c8009af0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:16.867 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:16.863+0000 7f84c2ffd700 1 -- 172.21.15.58:0/2954523915 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84c404ab10 con 0x7f84d40f55c0 2021-11-20T05:34:17.069 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:17.069+0000 7f84d8ead700 1 -- 172.21.15.58:0/2954523915 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f84d40fdd50 con 0x7f84d40f55c0 2021-11-20T05:34:17.069 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:17.069+0000 7f84c2ffd700 1 -- 172.21.15.58:0/2954523915 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 8 v8) v1 ==== 75+0+1466 (secure 0 0 0) 0x7f84c404e130 con 0x7f84d40f55c0 2021-11-20T05:34:17.070 INFO:teuthology.orchestra.run.smithi058.stdout:e8 2021-11-20T05:34:17.071 INFO:teuthology.orchestra.run.smithi058.stdout:enable_multiple, ever_enabled_multiple: 1,1 2021-11-20T05:34:17.071 INFO:teuthology.orchestra.run.smithi058.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T05:34:17.071 INFO:teuthology.orchestra.run.smithi058.stdout:legacy client fscid: 1 2021-11-20T05:34:17.072 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:34:17.072 INFO:teuthology.orchestra.run.smithi058.stdout:Filesystem 'cephfs' (1) 2021-11-20T05:34:17.072 INFO:teuthology.orchestra.run.smithi058.stdout:fs_name cephfs 2021-11-20T05:34:17.072 INFO:teuthology.orchestra.run.smithi058.stdout:epoch 8 2021-11-20T05:34:17.073 INFO:teuthology.orchestra.run.smithi058.stdout:flags 12 2021-11-20T05:34:17.073 INFO:teuthology.orchestra.run.smithi058.stdout:created 2021-11-20T05:34:06.306214+0000 2021-11-20T05:34:17.073 INFO:teuthology.orchestra.run.smithi058.stdout:modified 2021-11-20T05:34:15.893368+0000 2021-11-20T05:34:17.073 INFO:teuthology.orchestra.run.smithi058.stdout:tableserver 0 2021-11-20T05:34:17.074 INFO:teuthology.orchestra.run.smithi058.stdout:root 0 2021-11-20T05:34:17.074 INFO:teuthology.orchestra.run.smithi058.stdout:session_timeout 60 2021-11-20T05:34:17.075 INFO:teuthology.orchestra.run.smithi058.stdout:session_autoclose 300 2021-11-20T05:34:17.075 INFO:teuthology.orchestra.run.smithi058.stdout:max_file_size 1099511627776 2021-11-20T05:34:17.075 INFO:teuthology.orchestra.run.smithi058.stdout:required_client_features {} 2021-11-20T05:34:17.075 INFO:teuthology.orchestra.run.smithi058.stdout:last_failure 0 2021-11-20T05:34:17.076 INFO:teuthology.orchestra.run.smithi058.stdout:last_failure_osd_epoch 0 2021-11-20T05:34:17.076 INFO:teuthology.orchestra.run.smithi058.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T05:34:17.076 INFO:teuthology.orchestra.run.smithi058.stdout:max_mds 2 2021-11-20T05:34:17.077 INFO:teuthology.orchestra.run.smithi058.stdout:in 0,1 2021-11-20T05:34:17.077 INFO:teuthology.orchestra.run.smithi058.stdout:up {0=14428,1=14444} 2021-11-20T05:34:17.078 INFO:teuthology.orchestra.run.smithi058.stdout:failed 2021-11-20T05:34:17.078 INFO:teuthology.orchestra.run.smithi058.stdout:damaged 2021-11-20T05:34:17.079 INFO:teuthology.orchestra.run.smithi058.stdout:stopped 2021-11-20T05:34:17.079 INFO:teuthology.orchestra.run.smithi058.stdout:data_pools [3] 2021-11-20T05:34:17.079 INFO:teuthology.orchestra.run.smithi058.stdout:metadata_pool 2 2021-11-20T05:34:17.079 INFO:teuthology.orchestra.run.smithi058.stdout:inline_data disabled 2021-11-20T05:34:17.080 INFO:teuthology.orchestra.run.smithi058.stdout:balancer 2021-11-20T05:34:17.080 INFO:teuthology.orchestra.run.smithi058.stdout:standby_count_wanted 1 2021-11-20T05:34:17.080 INFO:teuthology.orchestra.run.smithi058.stdout:[mds.cephfs.smithi058.rxapzd{0:14428} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164]] 2021-11-20T05:34:17.080 INFO:teuthology.orchestra.run.smithi058.stdout:[mds.cephfs.smithi058.nuxpgf{1:14444} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.58:6828/4151830882,v1:172.21.15.58:6829/4151830882]] 2021-11-20T05:34:17.081 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:34:17.081 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:34:17.081 INFO:teuthology.orchestra.run.smithi058.stdout:Standby daemons: 2021-11-20T05:34:17.081 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:34:17.082 INFO:teuthology.orchestra.run.smithi058.stdout:[mds.cephfs.smithi187.wxduhu{-1:24259} state up:standby seq 2 addr [v2:172.21.15.187:6824/2274008158,v1:172.21.15.187:6825/2274008158]] 2021-11-20T05:34:17.083 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:17.071+0000 7f84d8ead700 1 -- 172.21.15.58:0/2954523915 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f84bc05c2b0 msgr2=0x7f84bc05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:17.084 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:17.071+0000 7f84d8ead700 1 --2- 172.21.15.58:0/2954523915 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f84bc05c2b0 0x7f84bc05e770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f84c8004ad0 tx=0x7f84c8009af0).stop 2021-11-20T05:34:17.084 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:17.071+0000 7f84d8ead700 1 -- 172.21.15.58:0/2954523915 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f84d40f55c0 msgr2=0x7f84d40f9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:17.084 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:17.071+0000 7f84d8ead700 1 --2- 172.21.15.58:0/2954523915 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f84d40f55c0 0x7f84d40f9f40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f84c4004e30 tx=0x7f84c4009b30).stop 2021-11-20T05:34:17.085 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:17.071+0000 7f84d8ead700 1 -- 172.21.15.58:0/2954523915 shutdown_connections 2021-11-20T05:34:17.085 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:17.071+0000 7f84d8ead700 1 --2- 172.21.15.58:0/2954523915 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f84bc05c2b0 0x7f84bc05e770 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:17.086 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:17.071+0000 7f84d8ead700 1 --2- 172.21.15.58:0/2954523915 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f84d40f55c0 0x7f84d40f9f40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:17.086 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:17.071+0000 7f84d8ead700 1 --2- 172.21.15.58:0/2954523915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f84d40f6860 0x7f84d40f83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:34:17.086 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:17.071+0000 7f84d8ead700 1 -- 172.21.15.58:0/2954523915 >> 172.21.15.58:0/2954523915 conn(0x7f84d40f0d40 msgr2=0x7f84d405f620 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:17.087 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:17.071+0000 7f84d8ead700 1 -- 172.21.15.58:0/2954523915 shutdown_connections 2021-11-20T05:34:17.087 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:17.071+0000 7f84d8ead700 1 -- 172.21.15.58:0/2954523915 wait complete. 2021-11-20T05:34:17.088 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 8 2021-11-20T05:34:17.197 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:16 smithi187 conmon[36358]: cluster 2021-11-20T05:34:15.893401+0000 mon.smithi058 (mon. 2021-11-20T05:34:17.197 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:16 smithi187 conmon[36358]: 0) 500 : cluster [INF] Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2021-11-20T05:34:17.198 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:16 smithi187 conmon[36358]: cluster 2021-11-20T05:34:15.893424+0000 mon.smithi058 (mon.0) 501 : cluster [INF] Cluster is now healthy 2021-11-20T05:34:17.198 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:16 smithi187 conmon[36358]: cluster 2021-11-20T05:34:15.896185+0000 mon.smithi058 (mon.0) 502 : cluster [DBG] mds.? [v2:172.21.15.58:6828/4151830882,v1:172.21.15.58:6829/4151830882] up:active 2021-11-20T05:34:17.198 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:16 smithi187 conmon[36358]: cluster 2021-11-20T05:34:15.896237+0000 mon.smithi058 (mon.0) 503 : cluster [DBG] mds.? [v2:172.21.15.187:6824/2274008158,v1:172.21.15.187:6825/2274008158] up:boot 2021-11-20T05:34:17.198 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:16 smithi187 conmon[36358]: cluster 2021-11-20T05:34:15.896306+0000 mon.smithi058 (mon.0) 504 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi058.rxapzd=up:active,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:34:17.199 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:16 smithi187 conmon[36358]: audit 2021-11-20T05:34:15.896415+0000 mon.smithi058 (mon.0) 505 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.wxduhu"}]: dispatch 2021-11-20T05:34:17.199 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:16 smithi187 conmon[36358]: cluster 2021-11-20T05:34:16.211106+0000 mgr.smithi058.zagdlk (mgr.14164) 197 : cluster [DBG] pgmap v159: 65 pgs: 65 active+clean; 1.7 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 4.5 KiB/s wr, 11 op/s 2021-11-20T05:34:17.199 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:16 smithi187 conmon[36358]: audit 2021-11-20T05:34:16.351384+0000 mon.smithi058 (mon.0) 506 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:17.199 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:16 smithi187 conmon[36358]: audit 2021-11-20T05:34:16.352050+0000 mon.smithi058 (mon.0) 507 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:34:17.219 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:16 smithi058 conmon[33537]: cluster 2021-11-20T05:34:15.893401+0000 2021-11-20T05:34:17.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:16 smithi058 conmon[33537]: mon.smithi058 (mon.0) 500 : cluster [INF] Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2021-11-20T05:34:17.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:16 smithi058 conmon[33537]: cluster 2021-11-20T05:34:15.893424+0000 mon.smithi058 (mon.0) 501 : cluster [INF] Cluster is now healthy 2021-11-20T05:34:17.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:16 smithi058 conmon[33537]: cluster 2021-11-20T05:34:15.896185+0000 mon.smithi058 (mon.0) 502 : cluster [DBG] mds.? [v2:172.21.15.58:6828/4151830882,v1:172.21.15.58:6829/4151830882] up:active 2021-11-20T05:34:17.220 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:16 smithi058 conmon[33537]: cluster 2021-11-20T05:34:15.896237+0000 mon.smithi058 (mon.0) 503 : cluster [DBG] mds.? [v2:172.21.15.187:6824/2274008158,v1:172.21.15.187:6825/2274008158] up:boot 2021-11-20T05:34:17.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:16 smithi058 conmon[33537]: cluster 2021-11-20T05:34:15.896306+0000 mon.smithi058 (mon.0) 504 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi058.rxapzd=up:active,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:34:17.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:16 smithi058 conmon[33537]: audit 2021-11-20T05:34:15.896415+0000 mon.smithi058 (mon.0) 505 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.wxduhu"}]: dispatch 2021-11-20T05:34:17.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:16 smithi058 conmon[33537]: cluster 2021-11-20T05:34:16.211106+0000 mgr.smithi058.zagdlk (mgr.14164) 197 : cluster [DBG] pgmap v159: 65 pgs: 65 active+clean; 1.7 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 4.5 KiB/s wr, 11 op/s 2021-11-20T05:34:17.221 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:16 smithi058 conmon[33537]: audit 2021-11-20T05:34:16.351384+0000 mon.smithi058 (mon.0) 506 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:17.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:16 smithi058 conmon[33537]: audit 2021-11-20T05:34:16. 2021-11-20T05:34:17.222 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:16 smithi058 conmon[33537]: 352050+0000 mon.smithi058 (mon.0) 507 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:34:17.495 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2021-11-20T05:34:18.197 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:17 smithi187 conmon[36358]: audit 2021-11-20T05:34:17.070164+0000 mon.smithi058 (mon.0) 508 : audit [DBG] from='client.? 172.21.15.58:0/2954523915' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-20T05:34:18.203 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:17 smithi058 conmon[33537]: audit 2021-11-20T05:34:17.070164+0000 mon.smithi058 (mon.0) 508 : audit [DBG] from='client.? 172.21.15.58:0/2954523915' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-20T05:34:19.177 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:18 smithi058 conmon[33537]: cluster 2021-11-20T05:34:19.177 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:18 smithi058 conmon[33537]: 2021-11-20T05:34:18.211527+0000 mgr.smithi058.zagdlk (mgr.14164) 198 : cluster [DBG] pgmap v160: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 4.2 KiB/s wr, 13 op/s 2021-11-20T05:34:19.197 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:18 smithi187 conmon[36358]: cluster 2021-11-20T05:34:18.211527+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:34:19.197 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:18 smithi187 conmon[36358]: 14164) 198 : cluster [DBG] pgmap v160: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 4.2 KiB/s wr, 13 op/s 2021-11-20T05:34:19.422 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.421+0000 7f5bd2028700 1 -- 172.21.15.58:0/574666644 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5bcc0f5430 msgr2=0x7f5bcc0f5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:19.422 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.421+0000 7f5bd2028700 1 --2- 172.21.15.58:0/574666644 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5bcc0f5430 0x7f5bcc0f5850 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f5bc8004660 tx=0x7f5bc8009b30).stop 2021-11-20T05:34:19.423 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.421+0000 7f5bd2028700 1 -- 172.21.15.58:0/574666644 shutdown_connections 2021-11-20T05:34:19.423 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.421+0000 7f5bd2028700 1 --2- 172.21.15.58:0/574666644 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bcc0f66d0 0x7f5bcc0f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:19.423 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.421+0000 7f5bd2028700 1 --2- 172.21.15.58:0/574666644 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5bcc0f5430 0x7f5bcc0f5850 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:19.423 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.421+0000 7f5bd2028700 1 -- 172.21.15.58:0/574666644 >> 172.21.15.58:0/574666644 conn(0x7f5bcc0f0b90 msgr2=0x7f5bcc0f2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:19.424 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.421+0000 7f5bd2028700 1 -- 172.21.15.58:0/574666644 shutdown_connections 2021-11-20T05:34:19.424 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.421+0000 7f5bd2028700 1 -- 172.21.15.58:0/574666644 wait complete. 2021-11-20T05:34:19.424 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.422+0000 7f5bd2028700 1 Processor -- start 2021-11-20T05:34:19.424 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.422+0000 7f5bd2028700 1 -- start start 2021-11-20T05:34:19.424 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.422+0000 7f5bd2028700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bcc0f5430 0x7f5bcc10bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:19.425 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.422+0000 7f5bd2028700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5bcc0f66d0 0x7f5bcc10e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:19.425 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.422+0000 7f5bd2028700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc0602b0 con 0x7f5bcc0f66d0 2021-11-20T05:34:19.425 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.422+0000 7f5bd2028700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc0603f0 con 0x7f5bcc0f5430 2021-11-20T05:34:19.425 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.422+0000 7f5bd0825700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5bcc0f66d0 0x7f5bcc10e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:19.426 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.422+0000 7f5bd0825700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5bcc0f66d0 0x7f5bcc10e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40568/0 (socket says 172.21.15.58:40568) 2021-11-20T05:34:19.426 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.422+0000 7f5bd0825700 1 -- 172.21.15.58:0/2396342532 learned_addr learned my addr 172.21.15.58:0/2396342532 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:19.426 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.422+0000 7f5bd1026700 1 --2- 172.21.15.58:0/2396342532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bcc0f5430 0x7f5bcc10bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:19.426 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.422+0000 7f5bd0825700 1 -- 172.21.15.58:0/2396342532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bcc0f5430 msgr2=0x7f5bcc10bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:19.426 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.422+0000 7f5bd0825700 1 --2- 172.21.15.58:0/2396342532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bcc0f5430 0x7f5bcc10bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:19.427 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.423+0000 7f5bd0825700 1 -- 172.21.15.58:0/2396342532 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc8005040 con 0x7f5bcc0f66d0 2021-11-20T05:34:19.427 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.423+0000 7f5bd0825700 1 --2- 172.21.15.58:0/2396342532 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5bcc0f66d0 0x7f5bcc10e3d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f5bbc00c790 tx=0x7f5bbc007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:19.428 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.423+0000 7f5bc1ffb700 1 -- 172.21.15.58:0/2396342532 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bbc002590 con 0x7f5bcc0f66d0 2021-11-20T05:34:19.429 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.423+0000 7f5bc1ffb700 1 -- 172.21.15.58:0/2396342532 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5bbc002ba0 con 0x7f5bcc0f66d0 2021-11-20T05:34:19.429 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.423+0000 7f5bc1ffb700 1 -- 172.21.15.58:0/2396342532 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bbc013d90 con 0x7f5bcc0f66d0 2021-11-20T05:34:19.429 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.423+0000 7f5bd2028700 1 -- 172.21.15.58:0/2396342532 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bcc10e8c0 con 0x7f5bcc0f66d0 2021-11-20T05:34:19.430 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.423+0000 7f5bd2028700 1 -- 172.21.15.58:0/2396342532 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bcc10ed40 con 0x7f5bcc0f66d0 2021-11-20T05:34:19.430 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.424+0000 7f5bd2028700 1 -- 172.21.15.58:0/2396342532 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bcc045ad0 con 0x7f5bcc0f66d0 2021-11-20T05:34:19.430 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.425+0000 7f5bc1ffb700 1 -- 172.21.15.58:0/2396342532 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5bbc01a060 con 0x7f5bcc0f66d0 2021-11-20T05:34:19.430 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.425+0000 7f5bc1ffb700 1 --2- 172.21.15.58:0/2396342532 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5bb805c300 0x7f5bb805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:19.431 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.426+0000 7f5bc1ffb700 1 -- 172.21.15.58:0/2396342532 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f5bbc076330 con 0x7f5bcc0f66d0 2021-11-20T05:34:19.431 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.426+0000 7f5bd1026700 1 --2- 172.21.15.58:0/2396342532 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5bb805c300 0x7f5bb805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:19.431 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.426+0000 7f5bd1026700 1 --2- 172.21.15.58:0/2396342532 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5bb805c300 0x7f5bb805e7c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f5bcc060040 tx=0x7f5bc8009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:19.432 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.429+0000 7f5bc1ffb700 1 -- 172.21.15.58:0/2396342532 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5bbc0490c0 con 0x7f5bcc0f66d0 2021-11-20T05:34:19.631 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.631+0000 7f5bd2028700 1 -- 172.21.15.58:0/2396342532 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bcc027a00 con 0x7f5bcc0f66d0 2021-11-20T05:34:19.632 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.632+0000 7f5bc1ffb700 1 -- 172.21.15.58:0/2396342532 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+2568 (secure 0 0 0) 0x7f5bbc04c6e0 con 0x7f5bcc0f66d0 2021-11-20T05:34:19.633 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.634+0000 7f5bd2028700 1 -- 172.21.15.58:0/2396342532 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5bb805c300 msgr2=0x7f5bb805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:19.634 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.634+0000 7f5bd2028700 1 --2- 172.21.15.58:0/2396342532 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5bb805c300 0x7f5bb805e7c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f5bcc060040 tx=0x7f5bc8009b30).stop 2021-11-20T05:34:19.634 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.634+0000 7f5bd2028700 1 -- 172.21.15.58:0/2396342532 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5bcc0f66d0 msgr2=0x7f5bcc10e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:19.634 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.634+0000 7f5bd2028700 1 --2- 172.21.15.58:0/2396342532 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5bcc0f66d0 0x7f5bcc10e3d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f5bbc00c790 tx=0x7f5bbc007d70).stop 2021-11-20T05:34:19.635 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.634+0000 7f5bd2028700 1 -- 172.21.15.58:0/2396342532 shutdown_connections 2021-11-20T05:34:19.635 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.634+0000 7f5bd2028700 1 --2- 172.21.15.58:0/2396342532 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5bb805c300 0x7f5bb805e7c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:19.635 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.634+0000 7f5bd2028700 1 --2- 172.21.15.58:0/2396342532 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bcc0f5430 0x7f5bcc10bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:19.636 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.634+0000 7f5bd2028700 1 --2- 172.21.15.58:0/2396342532 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5bcc0f66d0 0x7f5bcc10e3d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:19.636 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.634+0000 7f5bd2028700 1 -- 172.21.15.58:0/2396342532 >> 172.21.15.58:0/2396342532 conn(0x7f5bcc0f0b90 msgr2=0x7f5bcc0fa120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:19.636 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.634+0000 7f5bd2028700 1 -- 172.21.15.58:0/2396342532 shutdown_connections 2021-11-20T05:34:19.636 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:19.634+0000 7f5bd2028700 1 -- 172.21.15.58:0/2396342532 wait complete. 2021-11-20T05:34:19.637 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 8 2021-11-20T05:34:19.644 INFO:teuthology.orchestra.run.smithi058.stdout:true 2021-11-20T05:34:20.122 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c -- bash -c 'ceph --format=json mds versions | jq -e ". | add == 4"' 2021-11-20T05:34:20.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:20 smithi058 conmon[33537]: audit 2021-11-20T05:34: 2021-11-20T05:34:20.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:20 smithi058 conmon[33537]: 19.217561+0000 mon.smithi058 (mon.0) 509 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:20.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:20 smithi058 conmon[33537]: audit 2021-11-20T05:34:19.633089+0000 mon.smithi058 (mon.0) 510 : audit [DBG] from='client.? 172.21.15.58:0/2396342532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:34:20.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:20 smithi187 conmon[36358]: audit 2021-11-20T05:34:19.217561+0000 mon.smithi058 (mon.0) 509 2021-11-20T05:34:20.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:20 smithi187 conmon[36358]: : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:20.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:20 smithi187 conmon[36358]: audit 2021-11-20T05:34:19.633089+0000 mon.smithi058 (mon.0) 510 : audit [DBG] from='client.? 172.21.15.58:0/2396342532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:34:21.355 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:21 smithi058 conmon[33537]: cluster 2021-11-20T05:34:20.211979+0000 mgr.smithi058.zagdlk (mgr 2021-11-20T05:34:21.355 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:21 smithi058 conmon[33537]: .14164) 199 : cluster [DBG] pgmap v161: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 3.7 KiB/s wr, 12 op/s 2021-11-20T05:34:21.355 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:21 smithi058 conmon[33537]: cluster 2021-11-20T05:34:20.227836+0000 mon.smithi058 (mon.0) 511 : cluster [DBG] mds.? [v2:172.21.15.187:6824/2274008158,v1:172.21.15.187:6825/2274008158] up:standby 2021-11-20T05:34:21.356 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:21 smithi058 conmon[33537]: cluster 2021-11-20T05:34:20.228270+0000 mon.smithi058 (mon.0) 512 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi058.rxapzd=up:active,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:34:21.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:21 smithi187 conmon[36358]: cluster 2021-11-20T05:34:20.211979+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:34:21.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:21 smithi187 conmon[36358]: ) 199 : cluster [DBG] pgmap v161: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 3.7 KiB/s wr, 12 op/s 2021-11-20T05:34:21.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:21 smithi187 conmon[36358]: cluster 2021-11-20T05:34:20.227836+0000 mon.smithi058 (mon.0) 511 : cluster [DBG] mds.? [v2:172.21.15.187:6824/2274008158,v1:172.21.15.187:6825/2274008158] up:standby 2021-11-20T05:34:21.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:21 smithi187 conmon[36358]: cluster 2021-11-20T05:34:20.228270+0000 mon.smithi058 (mon.0) 512 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi058.rxapzd=up:active,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:34:22.268 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.264+0000 7ff94ab8f700 1 -- 172.21.15.58:0/2455718104 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff9440f66e0 msgr2=0x7ff9440f6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:22.268 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.264+0000 7ff94ab8f700 1 --2- 172.21.15.58:0/2455718104 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff9440f66e0 0x7ff9440f6b00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7ff938004660 tx=0x7ff938009b30).stop 2021-11-20T05:34:22.269 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.264+0000 7ff94ab8f700 1 -- 172.21.15.58:0/2455718104 shutdown_connections 2021-11-20T05:34:22.269 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.264+0000 7ff94ab8f700 1 --2- 172.21.15.58:0/2455718104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9440f7980 0x7ff9440f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:22.270 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.264+0000 7ff94ab8f700 1 --2- 172.21.15.58:0/2455718104 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff9440f66e0 0x7ff9440f6b00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:22.270 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.264+0000 7ff94ab8f700 1 -- 172.21.15.58:0/2455718104 >> 172.21.15.58:0/2455718104 conn(0x7ff9440f1e40 msgr2=0x7ff9440f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:22.270 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.264+0000 7ff94ab8f700 1 -- 172.21.15.58:0/2455718104 shutdown_connections 2021-11-20T05:34:22.271 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.264+0000 7ff94ab8f700 1 -- 172.21.15.58:0/2455718104 wait complete. 2021-11-20T05:34:22.271 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.265+0000 7ff94ab8f700 1 Processor -- start 2021-11-20T05:34:22.272 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.265+0000 7ff94ab8f700 1 -- start start 2021-11-20T05:34:22.272 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.265+0000 7ff94ab8f700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff9440f66e0 0x7ff944106f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:22.273 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.265+0000 7ff94ab8f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9440f7980 0x7ff944107480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:22.273 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.265+0000 7ff94ab8f700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff944107a30 con 0x7ff9440f66e0 2021-11-20T05:34:22.273 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.265+0000 7ff94ab8f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff944107b70 con 0x7ff9440f7980 2021-11-20T05:34:22.274 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.265+0000 7ff94892b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff9440f66e0 0x7ff944106f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:22.274 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.265+0000 7ff94892b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff9440f66e0 0x7ff944106f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40576/0 (socket says 172.21.15.58:40576) 2021-11-20T05:34:22.275 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.265+0000 7ff94892b700 1 -- 172.21.15.58:0/4136655122 learned_addr learned my addr 172.21.15.58:0/4136655122 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:22.275 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.265+0000 7ff94892b700 1 -- 172.21.15.58:0/4136655122 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9440f7980 msgr2=0x7ff944107480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:22.276 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.265+0000 7ff943fff700 1 --2- 172.21.15.58:0/4136655122 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9440f7980 0x7ff944107480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:22.276 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.265+0000 7ff94892b700 1 --2- 172.21.15.58:0/4136655122 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9440f7980 0x7ff944107480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:22.276 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.265+0000 7ff94892b700 1 -- 172.21.15.58:0/4136655122 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff938005040 con 0x7ff9440f66e0 2021-11-20T05:34:22.277 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.265+0000 7ff94892b700 1 --2- 172.21.15.58:0/4136655122 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff9440f66e0 0x7ff944106f90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7ff938004e30 tx=0x7ff938009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:22.277 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.266+0000 7ff9417fa700 1 -- 172.21.15.58:0/4136655122 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff938018070 con 0x7ff9440f66e0 2021-11-20T05:34:22.277 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.266+0000 7ff9417fa700 1 -- 172.21.15.58:0/4136655122 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff938013410 con 0x7ff9440f66e0 2021-11-20T05:34:22.278 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.266+0000 7ff9417fa700 1 -- 172.21.15.58:0/4136655122 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff93801c3c0 con 0x7ff9440f66e0 2021-11-20T05:34:22.278 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.266+0000 7ff94ab8f700 1 -- 172.21.15.58:0/4136655122 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9441084b0 con 0x7ff9440f66e0 2021-11-20T05:34:22.278 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.266+0000 7ff94ab8f700 1 -- 172.21.15.58:0/4136655122 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff944108a10 con 0x7ff9440f66e0 2021-11-20T05:34:22.278 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.267+0000 7ff94ab8f700 1 -- 172.21.15.58:0/4136655122 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff944045ad0 con 0x7ff9440f66e0 2021-11-20T05:34:22.279 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.267+0000 7ff9417fa700 1 -- 172.21.15.58:0/4136655122 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff93801d420 con 0x7ff9440f66e0 2021-11-20T05:34:22.279 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.267+0000 7ff9417fa700 1 --2- 172.21.15.58:0/4136655122 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff93405c2b0 0x7ff93405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:22.279 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.267+0000 7ff9417fa700 1 -- 172.21.15.58:0/4136655122 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7ff938077ac0 con 0x7ff9440f66e0 2021-11-20T05:34:22.279 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.268+0000 7ff943fff700 1 --2- 172.21.15.58:0/4136655122 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff93405c2b0 0x7ff93405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:22.279 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.268+0000 7ff943fff700 1 --2- 172.21.15.58:0/4136655122 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff93405c2b0 0x7ff93405e770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7ff930004a70 tx=0x7ff93000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:22.280 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.271+0000 7ff9417fa700 1 -- 172.21.15.58:0/4136655122 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff93804a780 con 0x7ff9440f66e0 2021-11-20T05:34:22.510 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.510+0000 7ff94ab8f700 1 -- 172.21.15.58:0/4136655122 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7ff944027a00 con 0x7ff9440f66e0 2021-11-20T05:34:22.511 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:22 smithi058 conmon[33537]: cluster 2021-11-20T05:34:21.230953+0000 2021-11-20T05:34:22.511 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:22 smithi058 conmon[33537]: mon.smithi058 (mon.0) 513 : cluster [DBG] mds.? [v2:172.21.15.187:6826/2997529975,v1:172.21.15.187:6827/2997529975] up:boot 2021-11-20T05:34:22.511 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:22 smithi058 conmon[33537]: cluster 2021-11-20T05:34:21.231006+0000 mon.smithi058 (mon.0) 514 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi058.rxapzd=up:active,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:34:22.512 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:22 smithi058 conmon[33537]: audit 2021-11-20T05:34:21.231099+0000 mon.smithi058 (mon.0) 515 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.fikisa"}]: dispatch 2021-11-20T05:34:22.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.511+0000 7ff9417fa700 1 -- 172.21.15.58:0/4136655122 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v10) v1 ==== 78+0+86 (secure 0 0 0) 0x7ff93804dda0 con 0x7ff9440f66e0 2021-11-20T05:34:22.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.512+0000 7ff94ab8f700 1 -- 172.21.15.58:0/4136655122 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff93405c2b0 msgr2=0x7ff93405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:22.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.512+0000 7ff94ab8f700 1 --2- 172.21.15.58:0/4136655122 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff93405c2b0 0x7ff93405e770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7ff930004a70 tx=0x7ff93000b040).stop 2021-11-20T05:34:22.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.512+0000 7ff94ab8f700 1 -- 172.21.15.58:0/4136655122 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff9440f66e0 msgr2=0x7ff944106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:22.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.512+0000 7ff94ab8f700 1 --2- 172.21.15.58:0/4136655122 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff9440f66e0 0x7ff944106f90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7ff938004e30 tx=0x7ff938009b30).stop 2021-11-20T05:34:22.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.513+0000 7ff94ab8f700 1 -- 172.21.15.58:0/4136655122 shutdown_connections 2021-11-20T05:34:22.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.513+0000 7ff94ab8f700 1 --2- 172.21.15.58:0/4136655122 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff93405c2b0 0x7ff93405e770 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:22.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.513+0000 7ff94ab8f700 1 --2- 172.21.15.58:0/4136655122 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff9440f66e0 0x7ff944106f90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:22.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.513+0000 7ff94ab8f700 1 --2- 172.21.15.58:0/4136655122 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9440f7980 0x7ff944107480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:22.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.513+0000 7ff94ab8f700 1 -- 172.21.15.58:0/4136655122 >> 172.21.15.58:0/4136655122 conn(0x7ff9440f1e40 msgr2=0x7ff9440fb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:22.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.513+0000 7ff94ab8f700 1 -- 172.21.15.58:0/4136655122 shutdown_connections 2021-11-20T05:34:22.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:22.513+0000 7ff94ab8f700 1 -- 172.21.15.58:0/4136655122 wait complete. 2021-11-20T05:34:22.523 INFO:teuthology.orchestra.run.smithi058.stdout:true 2021-11-20T05:34:22.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:22 smithi187 conmon[36358]: cluster 2021-11-20T05:34:21.230953+0000 mon.smithi058 (mon.0) 513 : cluster 2021-11-20T05:34:22.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:22 smithi187 conmon[36358]: [DBG] mds.? [v2:172.21.15.187:6826/2997529975,v1:172.21.15.187:6827/2997529975] up:boot 2021-11-20T05:34:22.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:22 smithi187 conmon[36358]: cluster 2021-11-20T05:34:21.231006+0000 mon.smithi058 (mon.0) 514 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi058.rxapzd=up:active,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:34:22.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:22 smithi187 conmon[36358]: audit 2021-11-20T05:34:21.231099 2021-11-20T05:34:22.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:22 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 515 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.fikisa"}]: dispatch 2021-11-20T05:34:22.915 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2021-11-20T05:34:22.939 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-11-20T05:34:23.144 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.136+0000 7f1d662c1700 1 -- 172.21.15.58:0/3996700142 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1d600f6840 msgr2=0x7f1d600f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:23.145 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.136+0000 7f1d662c1700 1 --2- 172.21.15.58:0/3996700142 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1d600f6840 0x7f1d600f6ca0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f1d50004660 tx=0x7f1d50009b30).stop 2021-11-20T05:34:23.145 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.136+0000 7f1d662c1700 1 -- 172.21.15.58:0/3996700142 shutdown_connections 2021-11-20T05:34:23.146 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.136+0000 7f1d662c1700 1 --2- 172.21.15.58:0/3996700142 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1d600f6840 0x7f1d600f6ca0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.146 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.136+0000 7f1d662c1700 1 --2- 172.21.15.58:0/3996700142 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d600f55a0 0x7f1d600f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.146 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.136+0000 7f1d662c1700 1 -- 172.21.15.58:0/3996700142 >> 172.21.15.58:0/3996700142 conn(0x7f1d600f0ce0 msgr2=0x7f1d600f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:23.147 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.136+0000 7f1d662c1700 1 -- 172.21.15.58:0/3996700142 shutdown_connections 2021-11-20T05:34:23.147 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.136+0000 7f1d662c1700 1 -- 172.21.15.58:0/3996700142 wait complete. 2021-11-20T05:34:23.148 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.137+0000 7f1d662c1700 1 Processor -- start 2021-11-20T05:34:23.148 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.137+0000 7f1d662c1700 1 -- start start 2021-11-20T05:34:23.149 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.137+0000 7f1d662c1700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1d600f55a0 0x7f1d60105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:23.149 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.137+0000 7f1d662c1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d600f6840 0x7f1d60106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:23.150 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.137+0000 7f1d662c1700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d60106840 con 0x7f1d600f55a0 2021-11-20T05:34:23.150 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.137+0000 7f1d662c1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d60106980 con 0x7f1d600f6840 2021-11-20T05:34:23.150 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.137+0000 7f1d5ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1d600f55a0 0x7f1d60105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:23.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.137+0000 7f1d5ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1d600f55a0 0x7f1d60105da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40586/0 (socket says 172.21.15.58:40586) 2021-11-20T05:34:23.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.137+0000 7f1d5ffff700 1 -- 172.21.15.58:0/3104361961 learned_addr learned my addr 172.21.15.58:0/3104361961 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:23.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.138+0000 7f1d5f7fe700 1 --2- 172.21.15.58:0/3104361961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d600f6840 0x7f1d60106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:23.152 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.138+0000 7f1d5ffff700 1 -- 172.21.15.58:0/3104361961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d600f6840 msgr2=0x7f1d60106290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:23.152 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.138+0000 7f1d5ffff700 1 --2- 172.21.15.58:0/3104361961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d600f6840 0x7f1d60106290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.152 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.138+0000 7f1d5ffff700 1 -- 172.21.15.58:0/3104361961 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d50005040 con 0x7f1d600f55a0 2021-11-20T05:34:23.153 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.138+0000 7f1d5f7fe700 1 --2- 172.21.15.58:0/3104361961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d600f6840 0x7f1d60106290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:34:23.153 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.138+0000 7f1d5ffff700 1 --2- 172.21.15.58:0/3104361961 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1d600f55a0 0x7f1d60105da0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f1d54002700 tx=0x7f1d54009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:23.154 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.138+0000 7f1d5cff9700 1 -- 172.21.15.58:0/3104361961 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d54009320 con 0x7f1d600f55a0 2021-11-20T05:34:23.154 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.138+0000 7f1d5cff9700 1 -- 172.21.15.58:0/3104361961 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d5400f040 con 0x7f1d600f55a0 2021-11-20T05:34:23.154 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.138+0000 7f1d662c1700 1 -- 172.21.15.58:0/3104361961 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6010b420 con 0x7f1d600f55a0 2021-11-20T05:34:23.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.138+0000 7f1d5cff9700 1 -- 172.21.15.58:0/3104361961 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d54014900 con 0x7f1d600f55a0 2021-11-20T05:34:23.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.138+0000 7f1d662c1700 1 -- 172.21.15.58:0/3104361961 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6010b9e0 con 0x7f1d600f55a0 2021-11-20T05:34:23.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.139+0000 7f1d662c1700 1 -- 172.21.15.58:0/3104361961 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d60027a00 con 0x7f1d600f55a0 2021-11-20T05:34:23.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.139+0000 7f1d5cff9700 1 -- 172.21.15.58:0/3104361961 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1d54009480 con 0x7f1d600f55a0 2021-11-20T05:34:23.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.140+0000 7f1d5cff9700 1 --2- 172.21.15.58:0/3104361961 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1d4805c2a0 0x7f1d4805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:23.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.140+0000 7f1d5cff9700 1 -- 172.21.15.58:0/3104361961 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f1d54076280 con 0x7f1d600f55a0 2021-11-20T05:34:23.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.140+0000 7f1d5f7fe700 1 --2- 172.21.15.58:0/3104361961 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1d4805c2a0 0x7f1d4805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:23.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.140+0000 7f1d5f7fe700 1 --2- 172.21.15.58:0/3104361961 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1d4805c2a0 0x7f1d4805e760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f1d50005010 tx=0x7f1d50009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:23.157 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.144+0000 7f1d5cff9700 1 -- 172.21.15.58:0/3104361961 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d54048f40 con 0x7f1d600f55a0 2021-11-20T05:34:23.309 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.310+0000 7f1d662c1700 1 -- 172.21.15.58:0/3104361961 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d600fdd30 con 0x7f1d600f55a0 2021-11-20T05:34:23.310 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.311+0000 7f1d5cff9700 1 -- 172.21.15.58:0/3104361961 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+14084 (secure 0 0 0) 0x7f1d5404c560 con 0x7f1d600f55a0 2021-11-20T05:34:23.310 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:34:23.311 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":42,"fsid":"b33f8c4a-49c2-11ec-8c2c-001a4aab830c","created":"2021-11-20T05:28:54.787342+0000","modified":"2021-11-20T05:34:08.318974+0000","last_up_change":"2021-11-20T05:33:10.122001+0000","last_in_change":"2021-11-20T05:32:53.577246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T05:32:27.068478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2021-11-20T05:34:04.784086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2021-11-20T05:34:05.738224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e2f5103a-e0d2-4bc3-88e1-f28420c98071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6802","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6803","nonce":2375873844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6804","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6805","nonce":2375873844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6808","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6809","nonce":2375873844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6806","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6807","nonce":2375873844}]},"public_addr":"172.21.15.58:6803/2375873844","cluster_addr":"172.21.15.58:6805/2375873844","heartbeat_back_addr":"172.21.15.58:6809/2375873844","heartbeat_front_addr":"172.21.15.58:6807/2375873844","state":["exists","up"]},{"osd":1,"uuid":"e0243e40-c76e-4c68-8ac0-8f555035e948","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6810","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6811","nonce":3875723869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6812","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6813","nonce":3875723869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6816","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6817","nonce":3875723869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6814","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6815","nonce":3875723869}]},"public_addr":"172.21.15.58:6811/3875723869","cluster_addr":"172.21.15.58:6813/3875723869","heartbeat_back_addr":"172.21.15.58:6817/3875723869","heartbeat_front_addr":"172.21.15.58:6815/3875723869","state":["exists","up"]},{"osd":2,"uuid":"9b35ec54-f60f-49c5-ab1b-d2c5b5eb72e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6818","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6819","nonce":745015076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6820","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6821","nonce":745015076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6824","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6825","nonce":745015076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6822","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6823","nonce":745015076}]},"public_addr":"172.21.15.58:6819/745015076","cluster_addr":"172.21.15.58:6821/745015076","heartbeat_back_addr":"172.21.15.58:6825/745015076","heartbeat_front_addr":"172.21.15.58:6823/745015076","state":["exists","up"]},{"osd":3,"uuid":"8390f146-b7e1-4039-83d3-7ed1c832dc8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6801","nonce":1941886381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6803","nonce":1941886381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6807","nonce":1941886381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6805","nonce":1941886381}]},"public_addr":"172.21.15.187:6801/1941886381","cluster_addr":"172.21.15.187:6803/1941886381","heartbeat_back_addr":"172.21.15.187:6807/1941886381","heartbeat_front_addr":"172.21.15.187:6805/1941886381","state":["exists","up"]},{"osd":4,"uuid":"89dc0c2f-c52c-48ca-b3ab-792e299d494d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6809","nonce":3483552299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6811","nonce":3483552299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6815","nonce":3483552299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6813","nonce":3483552299}]},"public_addr":"172.21.15.187:6809/3483552299","cluster_addr":"172.21.15.187:6811/3483552299","heartbeat_back_addr":"172.21.15.187:6815/3483552299","heartbeat_front_addr":"172.21.15.187:6813/3483552299","state":["exists","up"]},{"osd":5,"uuid":"8190ded3-7b78-4d71-9c23-e5bc4112fecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6817","nonce":566077514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6819","nonce":566077514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6823","nonce":566077514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6821","nonce":566077514}]},"public_addr":"172.21.15.187:6817/566077514","cluster_addr":"172.21.15.187:6819/566077514","heartbeat_back_addr":"172.21.15.187:6823/566077514","heartbeat_front_addr":"172.21.15.187:6821/566077514","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:31:38.522095+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:31:56.685873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:15.227272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:31.493769+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:50.070298+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:33:08.140441+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.58:6801/2422532259":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/1616356583":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/2030112153":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/3301163446":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6801/1261811851":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6800/1261811851":"2021-11-21T05:29:42.633679+0000","172.21.15.58:0/1447224424":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6800/324085383":"2021-11-21T05:29:22.266837+0000","172.21.15.58:6801/324085383":"2021-11-21T05:29:22.266837+0000","172.21.15.58:0/574129054":"2021-11-21T05:29:22.266837+0000","172.21.15.58:6800/2422532259":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/2571399354":"2021-11-21T05:29:22.266837+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T05:34:23.312 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.312+0000 7f1d662c1700 1 -- 172.21.15.58:0/3104361961 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1d4805c2a0 msgr2=0x7f1d4805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:23.313 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.312+0000 7f1d662c1700 1 --2- 172.21.15.58:0/3104361961 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1d4805c2a0 0x7f1d4805e760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f1d50005010 tx=0x7f1d50009b30).stop 2021-11-20T05:34:23.313 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.312+0000 7f1d662c1700 1 -- 172.21.15.58:0/3104361961 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1d600f55a0 msgr2=0x7f1d60105da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:23.313 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.312+0000 7f1d662c1700 1 --2- 172.21.15.58:0/3104361961 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1d600f55a0 0x7f1d60105da0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f1d54002700 tx=0x7f1d54009d80).stop 2021-11-20T05:34:23.313 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.313+0000 7f1d662c1700 1 -- 172.21.15.58:0/3104361961 shutdown_connections 2021-11-20T05:34:23.314 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.313+0000 7f1d662c1700 1 --2- 172.21.15.58:0/3104361961 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1d4805c2a0 0x7f1d4805e760 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.314 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.313+0000 7f1d662c1700 1 --2- 172.21.15.58:0/3104361961 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1d600f55a0 0x7f1d60105da0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.314 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.313+0000 7f1d662c1700 1 --2- 172.21.15.58:0/3104361961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d600f6840 0x7f1d60106290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.314 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.313+0000 7f1d662c1700 1 -- 172.21.15.58:0/3104361961 >> 172.21.15.58:0/3104361961 conn(0x7f1d600f0ce0 msgr2=0x7f1d600fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:23.314 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.313+0000 7f1d662c1700 1 -- 172.21.15.58:0/3104361961 shutdown_connections 2021-11-20T05:34:23.315 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.313+0000 7f1d662c1700 1 -- 172.21.15.58:0/3104361961 wait complete. 2021-11-20T05:34:23.325 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-20T05:32:27.068478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2021-11-20T05:34:04.784086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2021-11-20T05:34:05.738224+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-11-20T05:34:23.326 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2021-11-20T05:34:23.454 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.453+0000 7f4d1d209700 1 -- 172.21.15.58:0/1503675693 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4d180f5580 msgr2=0x7f4d180f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:23.455 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.453+0000 7f4d1d209700 1 --2- 172.21.15.58:0/1503675693 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4d180f5580 0x7f4d180f59a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f4d08004660 tx=0x7f4d08009b30).stop 2021-11-20T05:34:23.455 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.453+0000 7f4d1d209700 1 -- 172.21.15.58:0/1503675693 shutdown_connections 2021-11-20T05:34:23.455 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.453+0000 7f4d1d209700 1 --2- 172.21.15.58:0/1503675693 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d180f6820 0x7f4d180f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.455 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.453+0000 7f4d1d209700 1 --2- 172.21.15.58:0/1503675693 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4d180f5580 0x7f4d180f59a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.456 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.453+0000 7f4d1d209700 1 -- 172.21.15.58:0/1503675693 >> 172.21.15.58:0/1503675693 conn(0x7f4d180f0ce0 msgr2=0x7f4d180f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:23.456 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.453+0000 7f4d1d209700 1 -- 172.21.15.58:0/1503675693 shutdown_connections 2021-11-20T05:34:23.456 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.453+0000 7f4d1d209700 1 -- 172.21.15.58:0/1503675693 wait complete. 2021-11-20T05:34:23.456 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.453+0000 7f4d1d209700 1 Processor -- start 2021-11-20T05:34:23.457 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.453+0000 7f4d1d209700 1 -- start start 2021-11-20T05:34:23.457 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.454+0000 7f4d1d209700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d180f5580 0x7f4d18106e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:23.457 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.454+0000 7f4d1d209700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4d180f6820 0x7f4d18107300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:23.457 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.454+0000 7f4d1d209700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d181078b0 con 0x7f4d180f6820 2021-11-20T05:34:23.457 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.454+0000 7f4d1d209700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d181079f0 con 0x7f4d180f5580 2021-11-20T05:34:23.458 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.454+0000 7f4d1659c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4d180f6820 0x7f4d18107300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:23.458 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.454+0000 7f4d1659c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4d180f6820 0x7f4d18107300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40598/0 (socket says 172.21.15.58:40598) 2021-11-20T05:34:23.458 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.454+0000 7f4d1659c700 1 -- 172.21.15.58:0/1560867692 learned_addr learned my addr 172.21.15.58:0/1560867692 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:23.458 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.454+0000 7f4d16d9d700 1 --2- 172.21.15.58:0/1560867692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d180f5580 0x7f4d18106e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:23.459 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.454+0000 7f4d1659c700 1 -- 172.21.15.58:0/1560867692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d180f5580 msgr2=0x7f4d18106e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:23.459 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.454+0000 7f4d1659c700 1 --2- 172.21.15.58:0/1560867692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d180f5580 0x7f4d18106e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.459 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.454+0000 7f4d1659c700 1 -- 172.21.15.58:0/1560867692 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d08005040 con 0x7f4d180f6820 2021-11-20T05:34:23.459 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.454+0000 7f4d1659c700 1 --2- 172.21.15.58:0/1560867692 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4d180f6820 0x7f4d18107300 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f4d00002700 tx=0x7f4d00009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:23.459 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.454+0000 7f4d0f7fe700 1 -- 172.21.15.58:0/1560867692 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d00009320 con 0x7f4d180f6820 2021-11-20T05:34:23.460 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.454+0000 7f4d0f7fe700 1 -- 172.21.15.58:0/1560867692 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d0000f040 con 0x7f4d180f6820 2021-11-20T05:34:23.460 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.454+0000 7f4d1d209700 1 -- 172.21.15.58:0/1560867692 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1810c490 con 0x7f4d180f6820 2021-11-20T05:34:23.460 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.454+0000 7f4d0f7fe700 1 -- 172.21.15.58:0/1560867692 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d00014870 con 0x7f4d180f6820 2021-11-20T05:34:23.460 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.454+0000 7f4d1d209700 1 -- 172.21.15.58:0/1560867692 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1810ca50 con 0x7f4d180f6820 2021-11-20T05:34:23.461 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.455+0000 7f4d1d209700 1 -- 172.21.15.58:0/1560867692 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d180fdd10 con 0x7f4d180f6820 2021-11-20T05:34:23.462 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.455+0000 7f4d0f7fe700 1 -- 172.21.15.58:0/1560867692 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4d00009480 con 0x7f4d180f6820 2021-11-20T05:34:23.462 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.456+0000 7f4d0f7fe700 1 --2- 172.21.15.58:0/1560867692 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d0405c2a0 0x7f4d0405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:23.462 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.456+0000 7f4d0f7fe700 1 -- 172.21.15.58:0/1560867692 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f4d000761f0 con 0x7f4d180f6820 2021-11-20T05:34:23.463 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.456+0000 7f4d16d9d700 1 --2- 172.21.15.58:0/1560867692 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d0405c2a0 0x7f4d0405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:23.463 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.456+0000 7f4d16d9d700 1 --2- 172.21.15.58:0/1560867692 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d0405c2a0 0x7f4d0405e760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f4d08005010 tx=0x7f4d08009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:23.463 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.460+0000 7f4d0f7fe700 1 -- 172.21.15.58:0/1560867692 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d00048e50 con 0x7f4d180f6820 2021-11-20T05:34:23.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:23 smithi187 conmon[36358]: cluster 2021-11-20T05:34:22.212360+0000 mgr.smithi058.zagdlk (mgr.14164) 200 : cluster [DBG] 2021-11-20T05:34:23.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:23 smithi187 conmon[36358]: pgmap v162: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s wr, 9 op/s 2021-11-20T05:34:23.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:23 smithi187 conmon[36358]: audit 2021-11-20T05:34:22.511591+0000 mon.smithi058 (mon.0) 516 : audit [DBG] from='client.? 172.21.15.58:0/4136655122' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2021-11-20T05:34:23.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:23 smithi187 conmon[36358]: audit 2021-11-20T05:34:23.221865+0000 mon.smithi058 (mon 2021-11-20T05:34:23.579 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:23 smithi187 conmon[36358]: .0) 517 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:23.579 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:23 smithi187 conmon[36358]: audit 2021-11-20T05:34:23.225148+0000 2021-11-20T05:34:23.579 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:23 smithi187 conmon[36358]: mon.smithi058 (mon.0) 518 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:23.579 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:23 smithi187 conmon[36358]: audit 2021-11-20 2021-11-20T05:34:23.580 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:23 smithi187 conmon[36358]: T05:34:23.228856+0000 mon.smithi058 (mon 2021-11-20T05:34:23.580 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:23 smithi187 conmon[36358]: .0) 519 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:34:23.619 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.619+0000 7f4d1d209700 1 -- 172.21.15.58:0/1560867692 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f4d1810d150 con 0x7f4d180f6820 2021-11-20T05:34:23.620 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.620+0000 7f4d0f7fe700 1 -- 172.21.15.58:0/1560867692 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v42) v1 ==== 110+0+10 (secure 0 0 0) 0x7f4d0004c470 con 0x7f4d180f6820 2021-11-20T05:34:23.620 INFO:teuthology.orchestra.run.smithi058.stdout:pg_num: 1 2021-11-20T05:34:23.621 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.621+0000 7f4d1d209700 1 -- 172.21.15.58:0/1560867692 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d0405c2a0 msgr2=0x7f4d0405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:23.621 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.621+0000 7f4d1d209700 1 --2- 172.21.15.58:0/1560867692 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d0405c2a0 0x7f4d0405e760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f4d08005010 tx=0x7f4d08009b30).stop 2021-11-20T05:34:23.622 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.621+0000 7f4d1d209700 1 -- 172.21.15.58:0/1560867692 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4d180f6820 msgr2=0x7f4d18107300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:23.622 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.621+0000 7f4d1d209700 1 --2- 172.21.15.58:0/1560867692 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4d180f6820 0x7f4d18107300 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f4d00002700 tx=0x7f4d00009e00).stop 2021-11-20T05:34:23.622 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.621+0000 7f4d1d209700 1 -- 172.21.15.58:0/1560867692 shutdown_connections 2021-11-20T05:34:23.622 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.621+0000 7f4d1d209700 1 --2- 172.21.15.58:0/1560867692 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4d0405c2a0 0x7f4d0405e760 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.623 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.621+0000 7f4d1d209700 1 --2- 172.21.15.58:0/1560867692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4d180f5580 0x7f4d18106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.623 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.621+0000 7f4d1d209700 1 --2- 172.21.15.58:0/1560867692 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4d180f6820 0x7f4d18107300 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.623 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.621+0000 7f4d1d209700 1 -- 172.21.15.58:0/1560867692 >> 172.21.15.58:0/1560867692 conn(0x7f4d180f0ce0 msgr2=0x7f4d1805fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:23.623 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.621+0000 7f4d1d209700 1 -- 172.21.15.58:0/1560867692 shutdown_connections 2021-11-20T05:34:23.623 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.621+0000 7f4d1d209700 1 -- 172.21.15.58:0/1560867692 wait complete. 2021-11-20T05:34:23.632 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:23 smithi058 conmon[33537]: cluster 2021-11-20T05:34:22.212360+0000 mgr.smithi058.zagdlk (mgr.14164) 200 : cluster [DBG] pgmap v162: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s wr, 9 op/s 2021-11-20T05:34:23.632 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:23 smithi058 conmon[33537]: audit 2021-11-20T05:34:22.511591+0000 mon.smithi058 (mon.0) 516 : audit [DBG] from='client.? 172.21.15.58:0/4136655122' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2021-11-20T05:34:23.633 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:23 smithi058 conmon[33537]: audit 2021-11-20T05:34:23.221865+0000 mon.smithi058 (mon.0) 517 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:23.633 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:23 smithi058 conmon[33537]: audit 2021-11-20T05:34:23.225148+0000 mon.smithi058 (mon.0) 518 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:23.633 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:23 smithi058 conmon[33537]: audit 2021-11-20T05:34:23.228856+0000 mon.smithi058 (mon.0) 519 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:34:23.634 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2021-11-20T05:34:23.762 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.760+0000 7fa2f3ccf700 1 -- 172.21.15.58:0/3113155309 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa2ec0f6820 msgr2=0x7fa2ec0f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:23.762 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.760+0000 7fa2f3ccf700 1 --2- 172.21.15.58:0/3113155309 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa2ec0f6820 0x7fa2ec0f6c80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fa2dc004660 tx=0x7fa2dc009b30).stop 2021-11-20T05:34:23.762 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.760+0000 7fa2f3ccf700 1 -- 172.21.15.58:0/3113155309 shutdown_connections 2021-11-20T05:34:23.763 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.760+0000 7fa2f3ccf700 1 --2- 172.21.15.58:0/3113155309 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa2ec0f6820 0x7fa2ec0f6c80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.763 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.760+0000 7fa2f3ccf700 1 --2- 172.21.15.58:0/3113155309 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2ec0f5580 0x7fa2ec0f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.764 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.760+0000 7fa2f3ccf700 1 -- 172.21.15.58:0/3113155309 >> 172.21.15.58:0/3113155309 conn(0x7fa2ec0f0ce0 msgr2=0x7fa2ec0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:23.764 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.760+0000 7fa2f3ccf700 1 -- 172.21.15.58:0/3113155309 shutdown_connections 2021-11-20T05:34:23.765 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.760+0000 7fa2f3ccf700 1 -- 172.21.15.58:0/3113155309 wait complete. 2021-11-20T05:34:23.765 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.761+0000 7fa2f3ccf700 1 Processor -- start 2021-11-20T05:34:23.765 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.761+0000 7fa2f3ccf700 1 -- start start 2021-11-20T05:34:23.765 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.761+0000 7fa2f3ccf700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa2ec0f5580 0x7fa2ec106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:23.766 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.761+0000 7fa2f3ccf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2ec0f6820 0x7fa2ec107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:23.766 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.761+0000 7fa2f3ccf700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec107940 con 0x7fa2ec0f5580 2021-11-20T05:34:23.766 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.761+0000 7fa2f3ccf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec107a80 con 0x7fa2ec0f6820 2021-11-20T05:34:23.766 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.761+0000 7fa2f1a6b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa2ec0f5580 0x7fa2ec106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:23.766 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.761+0000 7fa2f1a6b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa2ec0f5580 0x7fa2ec106ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40606/0 (socket says 172.21.15.58:40606) 2021-11-20T05:34:23.767 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.761+0000 7fa2f1a6b700 1 -- 172.21.15.58:0/1030553963 learned_addr learned my addr 172.21.15.58:0/1030553963 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:23.767 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.761+0000 7fa2f126a700 1 --2- 172.21.15.58:0/1030553963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2ec0f6820 0x7fa2ec107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:23.767 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.761+0000 7fa2f1a6b700 1 -- 172.21.15.58:0/1030553963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2ec0f6820 msgr2=0x7fa2ec107390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:23.767 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.761+0000 7fa2f1a6b700 1 --2- 172.21.15.58:0/1030553963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2ec0f6820 0x7fa2ec107390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.768 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.761+0000 7fa2f1a6b700 1 -- 172.21.15.58:0/1030553963 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2dc005040 con 0x7fa2ec0f5580 2021-11-20T05:34:23.768 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.762+0000 7fa2f1a6b700 1 --2- 172.21.15.58:0/1030553963 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa2ec0f5580 0x7fa2ec106ea0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fa2e8002700 tx=0x7fa2e8009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:23.768 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.762+0000 7fa2e27fc700 1 -- 172.21.15.58:0/1030553963 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2e8009320 con 0x7fa2ec0f5580 2021-11-20T05:34:23.768 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.762+0000 7fa2e27fc700 1 -- 172.21.15.58:0/1030553963 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa2e800f040 con 0x7fa2ec0f5580 2021-11-20T05:34:23.768 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.762+0000 7fa2f3ccf700 1 -- 172.21.15.58:0/1030553963 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ec10c520 con 0x7fa2ec0f5580 2021-11-20T05:34:23.769 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.762+0000 7fa2e27fc700 1 -- 172.21.15.58:0/1030553963 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2e8014900 con 0x7fa2ec0f5580 2021-11-20T05:34:23.769 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.762+0000 7fa2f3ccf700 1 -- 172.21.15.58:0/1030553963 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ec10cae0 con 0x7fa2ec0f5580 2021-11-20T05:34:23.769 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.763+0000 7fa2f3ccf700 1 -- 172.21.15.58:0/1030553963 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ec0fdd10 con 0x7fa2ec0f5580 2021-11-20T05:34:23.770 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.763+0000 7fa2e27fc700 1 -- 172.21.15.58:0/1030553963 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa2e8009480 con 0x7fa2ec0f5580 2021-11-20T05:34:23.771 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.763+0000 7fa2e27fc700 1 --2- 172.21.15.58:0/1030553963 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa2d805c2a0 0x7fa2d805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:23.771 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.763+0000 7fa2e27fc700 1 -- 172.21.15.58:0/1030553963 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fa2e8076220 con 0x7fa2ec0f5580 2021-11-20T05:34:23.771 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.763+0000 7fa2f126a700 1 --2- 172.21.15.58:0/1030553963 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa2d805c2a0 0x7fa2d805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:23.771 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.764+0000 7fa2f126a700 1 --2- 172.21.15.58:0/1030553963 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa2d805c2a0 0x7fa2d805e760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa2dc004e30 tx=0x7fa2dc009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:23.772 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.767+0000 7fa2e27fc700 1 -- 172.21.15.58:0/1030553963 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa2e8048ee0 con 0x7fa2ec0f5580 2021-11-20T05:34:23.928 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.928+0000 7fa2f3ccf700 1 -- 172.21.15.58:0/1030553963 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fa2ec10c6b0 con 0x7fa2ec0f5580 2021-11-20T05:34:23.928 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.928+0000 7fa2e27fc700 1 -- 172.21.15.58:0/1030553963 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v42) v1 ==== 107+0+11 (secure 0 0 0) 0x7fa2ec10c6b0 con 0x7fa2ec0f5580 2021-11-20T05:34:23.928 INFO:teuthology.orchestra.run.smithi058.stdout:pg_num: 32 2021-11-20T05:34:23.930 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.930+0000 7fa2f3ccf700 1 -- 172.21.15.58:0/1030553963 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa2d805c2a0 msgr2=0x7fa2d805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:23.930 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.930+0000 7fa2f3ccf700 1 --2- 172.21.15.58:0/1030553963 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa2d805c2a0 0x7fa2d805e760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa2dc004e30 tx=0x7fa2dc009b30).stop 2021-11-20T05:34:23.930 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.930+0000 7fa2f3ccf700 1 -- 172.21.15.58:0/1030553963 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa2ec0f5580 msgr2=0x7fa2ec106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:23.930 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.930+0000 7fa2f3ccf700 1 --2- 172.21.15.58:0/1030553963 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa2ec0f5580 0x7fa2ec106ea0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fa2e8002700 tx=0x7fa2e8009d80).stop 2021-11-20T05:34:23.931 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.930+0000 7fa2f3ccf700 1 -- 172.21.15.58:0/1030553963 shutdown_connections 2021-11-20T05:34:23.931 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.930+0000 7fa2f3ccf700 1 --2- 172.21.15.58:0/1030553963 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa2d805c2a0 0x7fa2d805e760 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.931 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.930+0000 7fa2f3ccf700 1 --2- 172.21.15.58:0/1030553963 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa2ec0f5580 0x7fa2ec106ea0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.931 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.930+0000 7fa2f3ccf700 1 --2- 172.21.15.58:0/1030553963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2ec0f6820 0x7fa2ec107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:23.932 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.930+0000 7fa2f3ccf700 1 -- 172.21.15.58:0/1030553963 >> 172.21.15.58:0/1030553963 conn(0x7fa2ec0f0ce0 msgr2=0x7fa2ec05f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:23.932 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.930+0000 7fa2f3ccf700 1 -- 172.21.15.58:0/1030553963 shutdown_connections 2021-11-20T05:34:23.932 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:23.930+0000 7fa2f3ccf700 1 -- 172.21.15.58:0/1030553963 wait complete. 2021-11-20T05:34:23.942 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2021-11-20T05:34:24.070 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.068+0000 7f1566e13700 1 -- 172.21.15.58:0/1754055682 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15600f6b90 msgr2=0x7f15600f6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:24.070 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.068+0000 7f1566e13700 1 --2- 172.21.15.58:0/1754055682 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15600f6b90 0x7f15600f6fb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f1554004660 tx=0x7f1554009b30).stop 2021-11-20T05:34:24.071 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.068+0000 7f1566e13700 1 -- 172.21.15.58:0/1754055682 shutdown_connections 2021-11-20T05:34:24.071 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.068+0000 7f1566e13700 1 --2- 172.21.15.58:0/1754055682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15600f7530 0x7f15600f47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:24.071 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.068+0000 7f1566e13700 1 --2- 172.21.15.58:0/1754055682 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15600f6b90 0x7f15600f6fb0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:24.071 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.068+0000 7f1566e13700 1 -- 172.21.15.58:0/1754055682 >> 172.21.15.58:0/1754055682 conn(0x7f15600f04a0 msgr2=0x7f15600f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:24.072 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.068+0000 7f1566e13700 1 -- 172.21.15.58:0/1754055682 shutdown_connections 2021-11-20T05:34:24.072 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.068+0000 7f1566e13700 1 -- 172.21.15.58:0/1754055682 wait complete. 2021-11-20T05:34:24.072 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.068+0000 7f1566e13700 1 Processor -- start 2021-11-20T05:34:24.072 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.068+0000 7f1566e13700 1 -- start start 2021-11-20T05:34:24.072 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.069+0000 7f1566e13700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15600f6b90 0x7f1560109040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:24.073 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.069+0000 7f1566e13700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15600f7530 0x7f1560109530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:24.073 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.069+0000 7f1566e13700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1560109ae0 con 0x7f15600f6b90 2021-11-20T05:34:24.073 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.069+0000 7f1566e13700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1560109c20 con 0x7f15600f7530 2021-11-20T05:34:24.073 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.069+0000 7f1564baf700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15600f6b90 0x7f1560109040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:24.074 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.069+0000 7f1564baf700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15600f6b90 0x7f1560109040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40616/0 (socket says 172.21.15.58:40616) 2021-11-20T05:34:24.074 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.069+0000 7f1564baf700 1 -- 172.21.15.58:0/2937664618 learned_addr learned my addr 172.21.15.58:0/2937664618 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:24.074 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.069+0000 7f155ffff700 1 --2- 172.21.15.58:0/2937664618 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15600f7530 0x7f1560109530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:24.074 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.069+0000 7f1564baf700 1 -- 172.21.15.58:0/2937664618 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15600f7530 msgr2=0x7f1560109530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:24.074 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.069+0000 7f1564baf700 1 --2- 172.21.15.58:0/2937664618 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15600f7530 0x7f1560109530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:24.075 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.069+0000 7f1564baf700 1 -- 172.21.15.58:0/2937664618 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1554005040 con 0x7f15600f6b90 2021-11-20T05:34:24.075 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.069+0000 7f1564baf700 1 --2- 172.21.15.58:0/2937664618 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15600f6b90 0x7f1560109040 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f1554009f50 tx=0x7f1554009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:24.075 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.069+0000 7f155d7fa700 1 -- 172.21.15.58:0/2937664618 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1554018070 con 0x7f15600f6b90 2021-11-20T05:34:24.075 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.069+0000 7f1566e13700 1 -- 172.21.15.58:0/2937664618 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f156010e660 con 0x7f15600f6b90 2021-11-20T05:34:24.076 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.069+0000 7f155d7fa700 1 -- 172.21.15.58:0/2937664618 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1554013410 con 0x7f15600f6b90 2021-11-20T05:34:24.076 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.069+0000 7f155d7fa700 1 -- 172.21.15.58:0/2937664618 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f155401c3c0 con 0x7f15600f6b90 2021-11-20T05:34:24.076 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.069+0000 7f1566e13700 1 -- 172.21.15.58:0/2937664618 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f156010ec20 con 0x7f15600f6b90 2021-11-20T05:34:24.076 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.070+0000 7f1566e13700 1 -- 172.21.15.58:0/2937664618 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1560027a00 con 0x7f15600f6b90 2021-11-20T05:34:24.078 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.071+0000 7f155d7fa700 1 -- 172.21.15.58:0/2937664618 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f155401d420 con 0x7f15600f6b90 2021-11-20T05:34:24.078 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.071+0000 7f155d7fa700 1 --2- 172.21.15.58:0/2937664618 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f155005c2a0 0x7f155005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:24.078 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.071+0000 7f155d7fa700 1 -- 172.21.15.58:0/2937664618 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f1554077b40 con 0x7f15600f6b90 2021-11-20T05:34:24.079 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.071+0000 7f155ffff700 1 --2- 172.21.15.58:0/2937664618 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f155005c2a0 0x7f155005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:24.079 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.072+0000 7f155ffff700 1 --2- 172.21.15.58:0/2937664618 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f155005c2a0 0x7f155005e760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f154c004c40 tx=0x7f154c00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:24.079 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.075+0000 7f155d7fa700 1 -- 172.21.15.58:0/2937664618 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f155404a800 con 0x7f15600f6b90 2021-11-20T05:34:24.244 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.245+0000 7f1566e13700 1 -- 172.21.15.58:0/2937664618 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f1560101080 con 0x7f15600f6b90 2021-11-20T05:34:24.245 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.245+0000 7f155d7fa700 1 -- 172.21.15.58:0/2937664618 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v42) v1 ==== 107+0+11 (secure 0 0 0) 0x7f155404de20 con 0x7f15600f6b90 2021-11-20T05:34:24.245 INFO:teuthology.orchestra.run.smithi058.stdout:pg_num: 32 2021-11-20T05:34:24.246 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.247+0000 7f1566e13700 1 -- 172.21.15.58:0/2937664618 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f155005c2a0 msgr2=0x7f155005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:24.247 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.247+0000 7f1566e13700 1 --2- 172.21.15.58:0/2937664618 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f155005c2a0 0x7f155005e760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f154c004c40 tx=0x7f154c00b040).stop 2021-11-20T05:34:24.247 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.247+0000 7f1566e13700 1 -- 172.21.15.58:0/2937664618 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15600f6b90 msgr2=0x7f1560109040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:24.247 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.247+0000 7f1566e13700 1 --2- 172.21.15.58:0/2937664618 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15600f6b90 0x7f1560109040 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f1554009f50 tx=0x7f1554009b30).stop 2021-11-20T05:34:24.247 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.247+0000 7f1566e13700 1 -- 172.21.15.58:0/2937664618 shutdown_connections 2021-11-20T05:34:24.248 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.247+0000 7f1566e13700 1 --2- 172.21.15.58:0/2937664618 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f155005c2a0 0x7f155005e760 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:24.248 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.247+0000 7f1566e13700 1 --2- 172.21.15.58:0/2937664618 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15600f6b90 0x7f1560109040 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:24.248 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.247+0000 7f1566e13700 1 --2- 172.21.15.58:0/2937664618 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15600f7530 0x7f1560109530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:24.248 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.247+0000 7f1566e13700 1 -- 172.21.15.58:0/2937664618 >> 172.21.15.58:0/2937664618 conn(0x7f15600f04a0 msgr2=0x7f15600fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:24.249 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.247+0000 7f1566e13700 1 -- 172.21.15.58:0/2937664618 shutdown_connections 2021-11-20T05:34:24.249 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.247+0000 7f1566e13700 1 -- 172.21.15.58:0/2937664618 wait complete. 2021-11-20T05:34:24.259 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:34:24.383 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.381+0000 7f55d562d700 1 -- 172.21.15.58:0/1978476366 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55d00f6840 msgr2=0x7f55d00f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:24.383 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.381+0000 7f55d562d700 1 --2- 172.21.15.58:0/1978476366 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55d00f6840 0x7f55d00f6ca0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f55c4004660 tx=0x7f55c4009b30).stop 2021-11-20T05:34:24.384 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.381+0000 7f55d562d700 1 -- 172.21.15.58:0/1978476366 shutdown_connections 2021-11-20T05:34:24.384 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.381+0000 7f55d562d700 1 --2- 172.21.15.58:0/1978476366 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55d00f6840 0x7f55d00f6ca0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:24.384 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.381+0000 7f55d562d700 1 --2- 172.21.15.58:0/1978476366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55d00f55a0 0x7f55d00f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:24.384 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.381+0000 7f55d562d700 1 -- 172.21.15.58:0/1978476366 >> 172.21.15.58:0/1978476366 conn(0x7f55d00f0ce0 msgr2=0x7f55d00f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:24.385 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.381+0000 7f55d562d700 1 -- 172.21.15.58:0/1978476366 shutdown_connections 2021-11-20T05:34:24.385 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.381+0000 7f55d562d700 1 -- 172.21.15.58:0/1978476366 wait complete. 2021-11-20T05:34:24.385 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.382+0000 7f55d562d700 1 Processor -- start 2021-11-20T05:34:24.385 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.382+0000 7f55d562d700 1 -- start start 2021-11-20T05:34:24.386 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.382+0000 7f55d562d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55d00f55a0 0x7f55d005ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:24.386 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.382+0000 7f55d562d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55d00f6840 0x7f55d010bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:24.386 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.382+0000 7f55d562d700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d010c180 con 0x7f55d00f6840 2021-11-20T05:34:24.386 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.382+0000 7f55d562d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d010c2c0 con 0x7f55d00f55a0 2021-11-20T05:34:24.387 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.382+0000 7f55ce7fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55d00f6840 0x7f55d010bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:24.387 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.382+0000 7f55ce7fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55d00f6840 0x7f55d010bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40628/0 (socket says 172.21.15.58:40628) 2021-11-20T05:34:24.387 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.382+0000 7f55ce7fc700 1 -- 172.21.15.58:0/1841370651 learned_addr learned my addr 172.21.15.58:0/1841370651 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:24.387 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.382+0000 7f55ceffd700 1 --2- 172.21.15.58:0/1841370651 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55d00f55a0 0x7f55d005ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:24.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.383+0000 7f55ce7fc700 1 -- 172.21.15.58:0/1841370651 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55d00f55a0 msgr2=0x7f55d005ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:24.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.383+0000 7f55ce7fc700 1 --2- 172.21.15.58:0/1841370651 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55d00f55a0 0x7f55d005ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:24.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.383+0000 7f55ce7fc700 1 -- 172.21.15.58:0/1841370651 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c4005040 con 0x7f55d00f6840 2021-11-20T05:34:24.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.383+0000 7f55ce7fc700 1 --2- 172.21.15.58:0/1841370651 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55d00f6840 0x7f55d010bc90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f55c4009d30 tx=0x7f55c4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:24.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.383+0000 7f55b77fe700 1 -- 172.21.15.58:0/1841370651 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55c401c070 con 0x7f55d00f6840 2021-11-20T05:34:24.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.383+0000 7f55d562d700 1 -- 172.21.15.58:0/1841370651 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d0105a60 con 0x7f55d00f6840 2021-11-20T05:34:24.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.383+0000 7f55b77fe700 1 -- 172.21.15.58:0/1841370651 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55c4013410 con 0x7f55d00f6840 2021-11-20T05:34:24.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.383+0000 7f55b77fe700 1 -- 172.21.15.58:0/1841370651 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55c4021780 con 0x7f55d00f6840 2021-11-20T05:34:24.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.383+0000 7f55d562d700 1 -- 172.21.15.58:0/1841370651 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d0105fc0 con 0x7f55d00f6840 2021-11-20T05:34:24.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.384+0000 7f55d562d700 1 -- 172.21.15.58:0/1841370651 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d0027a00 con 0x7f55d00f6840 2021-11-20T05:34:24.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.384+0000 7f55b77fe700 1 -- 172.21.15.58:0/1841370651 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f55c40219c0 con 0x7f55d00f6840 2021-11-20T05:34:24.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.385+0000 7f55b77fe700 1 --2- 172.21.15.58:0/1841370651 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f55b805c2a0 0x7f55b805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:24.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.385+0000 7f55b77fe700 1 -- 172.21.15.58:0/1841370651 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f55c407c040 con 0x7f55d00f6840 2021-11-20T05:34:24.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.385+0000 7f55ceffd700 1 --2- 172.21.15.58:0/1841370651 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f55b805c2a0 0x7f55b805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:24.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.385+0000 7f55ceffd700 1 --2- 172.21.15.58:0/1841370651 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f55b805c2a0 0x7f55b805e760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f55d00f66a0 tx=0x7f55c000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:24.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.389+0000 7f55b77fe700 1 -- 172.21.15.58:0/1841370651 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f55c404ed00 con 0x7f55d00f6840 2021-11-20T05:34:24.609 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.609+0000 7f55d562d700 1 -- 172.21.15.58:0/1841370651 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55d00fdd30 con 0x7f55d00f6840 2021-11-20T05:34:24.610 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.610+0000 7f55b77fe700 1 -- 172.21.15.58:0/1841370651 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+2947 (secure 0 0 0) 0x7f55c4052320 con 0x7f55d00f6840 2021-11-20T05:34:24.611 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:34:24.611 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24259,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2274008158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2274008158},{"type":"v1","addr":"172.21.15.187:6825","nonce":2274008158}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":9},{"gid":24271,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/2997529975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2997529975},{"type":"v1","addr":"172.21.15.187:6827","nonce":2997529975}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:34:15.893368+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14428,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14428":{"gid":14428,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.58:6827/3420124164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3420124164},{"type":"v1","addr":"172.21.15.58:6827","nonce":3420124164}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:34:24.612 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.612+0000 7f55d562d700 1 -- 172.21.15.58:0/1841370651 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f55b805c2a0 msgr2=0x7f55b805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:24.613 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.612+0000 7f55d562d700 1 --2- 172.21.15.58:0/1841370651 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f55b805c2a0 0x7f55b805e760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f55d00f66a0 tx=0x7f55c000b040).stop 2021-11-20T05:34:24.613 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.612+0000 7f55d562d700 1 -- 172.21.15.58:0/1841370651 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55d00f6840 msgr2=0x7f55d010bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:24.614 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.612+0000 7f55d562d700 1 --2- 172.21.15.58:0/1841370651 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55d00f6840 0x7f55d010bc90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f55c4009d30 tx=0x7f55c4009b30).stop 2021-11-20T05:34:24.614 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.612+0000 7f55d562d700 1 -- 172.21.15.58:0/1841370651 shutdown_connections 2021-11-20T05:34:24.614 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.612+0000 7f55d562d700 1 --2- 172.21.15.58:0/1841370651 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f55b805c2a0 0x7f55b805e760 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:24.614 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.612+0000 7f55d562d700 1 --2- 172.21.15.58:0/1841370651 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55d00f55a0 0x7f55d005ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:24.615 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.612+0000 7f55d562d700 1 --2- 172.21.15.58:0/1841370651 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55d00f6840 0x7f55d010bc90 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:24.615 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.612+0000 7f55d562d700 1 -- 172.21.15.58:0/1841370651 >> 172.21.15.58:0/1841370651 conn(0x7f55d00f0ce0 msgr2=0x7f55d00fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:24.615 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.612+0000 7f55d562d700 1 -- 172.21.15.58:0/1841370651 shutdown_connections 2021-11-20T05:34:24.615 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:24.612+0000 7f55d562d700 1 -- 172.21.15.58:0/1841370651 wait complete. 2021-11-20T05:34:24.616 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 10 2021-11-20T05:34:24.622 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:24 smithi058 conmon[33537]: audit 2021-11-20T05:34:23.311295+0000 mon.smithi058 (mon.0) 520 : audit 2021-11-20T05:34:24.623 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:24 smithi058 conmon[33537]: [DBG] from='client.? 172.21.15.58:0/3104361961' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T05:34:24.623 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:24 smithi058 conmon[33537]: audit 2021-11-20T05:34:23.620868+0000 mon.smithi058 (mon.0) 521 : audit [DBG] from='client.? 172.21.15.58:0/1560867692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T05:34:24.623 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:24 smithi058 conmon[33537]: audit 2021-11-20T05:34:23.929344+0000 mon.smithi058 (mon.0) 522 : audit [DBG] from='client.? 172.21.15.58:0/1030553963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T05:34:24.624 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 8, 'max_mds': 2, 'flags': 18} 2021-11-20T05:34:24.625 INFO:teuthology.run_tasks:Running task kclient... 2021-11-20T05:34:24.707 INFO:tasks.kclient:Mounting kernel clients... 2021-11-20T05:34:24.708 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2021-11-20T05:34:24.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:24.709 DEBUG:teuthology.orchestra.run.smithi058:> ip netns list 2021-11-20T05:34:24.710 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:24 smithi187 conmon[36358]: audit 2021-11-20T05:34:23.311295+0000 mon.smithi058 (mon. 2021-11-20T05:34:24.710 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:24 smithi187 conmon[36358]: 0) 520 : audit [DBG] from='client.? 172.21.15.58:0/3104361961' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T05:34:24.710 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:24 smithi187 conmon[36358]: audit 2021-11-20T05:34:23.620868+0000 mon.smithi058 (mon.0) 521 : audit [DBG] from='client.? 172.21.15.58:0/1560867692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T05:34:24.711 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:24 smithi187 conmon[36358]: audit 2021-11-20T05:34:23.929344+0000 mon.smithi058 (mon.0) 522 : audit [DBG] from='client.? 172.21.15.58:0/1030553963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T05:34:24.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:24.729 DEBUG:teuthology.orchestra.run.smithi058:> sudo ip link delete ceph-brx 2021-11-20T05:34:24.798 INFO:teuthology.orchestra.run.smithi058.stderr:Cannot find device "ceph-brx" 2021-11-20T05:34:24.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T05:34:24.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:24.800 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list 2021-11-20T05:34:24.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:24.820 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link delete ceph-brx 2021-11-20T05:34:24.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T05:34:24.892 INFO:teuthology.orchestra.run.smithi187.stderr:Cannot find device "ceph-brx" 2021-11-20T05:34:24.892 INFO:tasks.kclient:client.0 config is {} 2021-11-20T05:34:24.893 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2021-11-20T05:34:24.893 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi058.front.sepia.ceph.com 2021-11-20T05:34:24.894 INFO:tasks.cephfs.mount:self.client.name = client.0 2021-11-20T05:34:24.894 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2021-11-20T05:34:24.894 INFO:tasks.cephfs.mount:self.cephfs_name = None 2021-11-20T05:34:24.895 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2021-11-20T05:34:24.895 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2021-11-20T05:34:24.895 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2021-11-20T05:34:24.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:24.896 DEBUG:teuthology.orchestra.run.smithi058:> ip addr 2021-11-20T05:34:24.913 INFO:teuthology.orchestra.run.smithi058.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2021-11-20T05:34:24.914 INFO:teuthology.orchestra.run.smithi058.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2021-11-20T05:34:24.914 INFO:teuthology.orchestra.run.smithi058.stdout: inet 127.0.0.1/8 scope host lo 2021-11-20T05:34:24.914 INFO:teuthology.orchestra.run.smithi058.stdout: valid_lft forever preferred_lft forever 2021-11-20T05:34:24.914 INFO:teuthology.orchestra.run.smithi058.stdout: inet6 ::1/128 scope host 2021-11-20T05:34:24.914 INFO:teuthology.orchestra.run.smithi058.stdout: valid_lft forever preferred_lft forever 2021-11-20T05:34:24.915 INFO:teuthology.orchestra.run.smithi058.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T05:34:24.915 INFO:teuthology.orchestra.run.smithi058.stdout: link/ether 0c:c4:7a:6b:ff:7c brd ff:ff:ff:ff:ff:ff 2021-11-20T05:34:24.915 INFO:teuthology.orchestra.run.smithi058.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2021-11-20T05:34:24.915 INFO:teuthology.orchestra.run.smithi058.stdout: link/ether ac:1f:6b:a5:c8:e4 brd ff:ff:ff:ff:ff:ff 2021-11-20T05:34:24.916 INFO:teuthology.orchestra.run.smithi058.stdout: inet 172.21.15.58/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2021-11-20T05:34:24.916 INFO:teuthology.orchestra.run.smithi058.stdout: valid_lft 42220sec preferred_lft 42220sec 2021-11-20T05:34:24.916 INFO:teuthology.orchestra.run.smithi058.stdout: inet6 fe80::ae1f:6bff:fea5:c8e4/64 scope link noprefixroute 2021-11-20T05:34:24.916 INFO:teuthology.orchestra.run.smithi058.stdout: valid_lft forever preferred_lft forever 2021-11-20T05:34:24.917 INFO:teuthology.orchestra.run.smithi058.stdout:4: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T05:34:24.917 INFO:teuthology.orchestra.run.smithi058.stdout: link/ether 0c:c4:7a:6b:ff:7d brd ff:ff:ff:ff:ff:ff 2021-11-20T05:34:24.917 INFO:teuthology.orchestra.run.smithi058.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T05:34:24.917 INFO:teuthology.orchestra.run.smithi058.stdout: link/ether ac:1f:6b:a5:c8:e5 brd ff:ff:ff:ff:ff:ff 2021-11-20T05:34:24.918 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2021-11-20T05:34:24.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:24.919 DEBUG:teuthology.orchestra.run.smithi058:> (cd / && exec bash -c ' 2021-11-20T05:34:24.919 DEBUG:teuthology.orchestra.run.smithi058:> set -e 2021-11-20T05:34:24.919 DEBUG:teuthology.orchestra.run.smithi058:> sudo ip link add name ceph-brx type bridge 2021-11-20T05:34:24.920 DEBUG:teuthology.orchestra.run.smithi058:> sudo ip addr flush dev ceph-brx 2021-11-20T05:34:24.920 DEBUG:teuthology.orchestra.run.smithi058:> sudo ip link set ceph-brx up 2021-11-20T05:34:24.920 DEBUG:teuthology.orchestra.run.smithi058:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2021-11-20T05:34:24.921 DEBUG:teuthology.orchestra.run.smithi058:> ') 2021-11-20T05:34:25.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:25.149 DEBUG:teuthology.orchestra.run.smithi058:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2021-11-20T05:34:25.177 INFO:teuthology.orchestra.run.smithi058.stdout:1 2021-11-20T05:34:25.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:25.178 DEBUG:teuthology.orchestra.run.smithi058:> route 2021-11-20T05:34:25.268 INFO:teuthology.orchestra.run.smithi058.stdout:Kernel IP routing table 2021-11-20T05:34:25.268 INFO:teuthology.orchestra.run.smithi058.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2021-11-20T05:34:25.268 INFO:teuthology.orchestra.run.smithi058.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2021-11-20T05:34:25.268 INFO:teuthology.orchestra.run.smithi058.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2021-11-20T05:34:25.269 INFO:teuthology.orchestra.run.smithi058.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2021-11-20T05:34:25.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:25.270 DEBUG:teuthology.orchestra.run.smithi058:> (cd / && exec bash -c ' 2021-11-20T05:34:25.270 DEBUG:teuthology.orchestra.run.smithi058:> set -e 2021-11-20T05:34:25.270 DEBUG:teuthology.orchestra.run.smithi058:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2021-11-20T05:34:25.271 DEBUG:teuthology.orchestra.run.smithi058:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2021-11-20T05:34:25.271 DEBUG:teuthology.orchestra.run.smithi058:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2021-11-20T05:34:25.271 DEBUG:teuthology.orchestra.run.smithi058:> ') 2021-11-20T05:34:25.309 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:25 smithi058 conmon[33537]: cluster 2021-11-20T05:34:24.212754+0000 mgr.smithi058.zagdlk (mgr.14164) 201 : cluster [DBG] pgmap v163: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s wr, 7 op/s 2021-11-20T05:34:25.310 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:25 smithi058 conmon[33537]: audit 2021-11-20T05:34:24.246131+0000 mon.smithi058 (mon.0) 523 : audit [DBG] from='client.? 172.21.15.58:0/2937664618' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T05:34:25.311 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:25 smithi058 conmon[33537]: audit 2021-11-20T05:34:24.611041+0000 mon.smithi058 (mon.0) 524 : audit [DBG] from='client.? 172.21.15.58:0/1841370651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:34:25.311 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:25 smithi058 conmon[33537]: cluster 2021-11-20T05:34:25.224113+0000 mon.smithi058 (mon.0) 525 : cluster [DBG] mds.? [v2:172.21.15.187:6826/2997529975,v1:172.21.15.187:6827/2997529975] up:standby 2021-11-20T05:34:25.311 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:25 smithi058 conmon[33537]: cluster 2021-11-20T05:34:25.224186+0000 mon.smithi058 (mon.0) 526 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi058.rxapzd=up:active,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:34:25.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:25.507 DEBUG:teuthology.orchestra.run.smithi058:> ip netns list 2021-11-20T05:34:25.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:25.564 DEBUG:teuthology.orchestra.run.smithi058:> ip netns list-id 2021-11-20T05:34:25.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:25.619 DEBUG:teuthology.orchestra.run.smithi058:> (cd / && exec bash -c ' 2021-11-20T05:34:25.619 DEBUG:teuthology.orchestra.run.smithi058:> set -e 2021-11-20T05:34:25.620 DEBUG:teuthology.orchestra.run.smithi058:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2021-11-20T05:34:25.620 DEBUG:teuthology.orchestra.run.smithi058:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2021-11-20T05:34:25.620 DEBUG:teuthology.orchestra.run.smithi058:> ') 2021-11-20T05:34:25.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:25 smithi187 conmon[36358]: cluster 2021-11-20T05:34:24.212754+0000 mgr.smithi058.zagdlk (mgr.14164) 201 : 2021-11-20T05:34:25.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:25 smithi187 conmon[36358]: cluster [DBG] pgmap v163: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s wr, 7 op/s 2021-11-20T05:34:25.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:25 smithi187 conmon[36358]: audit 2021-11-20T05:34:24.246131+0000 mon.smithi058 (mon.0) 523 : audit [DBG] from='client.? 172.21.15.58:0/2937664618' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T05:34:25.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:25 smithi187 conmon[36358]: audit 2021-11-20T05:34:24.611041+0000 mon.smithi058 (mon.0) 524 : audit [DBG] from='client.? 172.21.15.58:0/1841370651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:34:25.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:25 smithi187 conmon[36358]: cluster 2021-11-20T05:34:25.224113+0000 mon.smithi058 (mon.0) 525 : cluster [DBG] mds.? [v2:172.21.15.187:6826/2997529975,v1:172.21.15.187:6827/2997529975] up:standby 2021-11-20T05:34:25.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:25 smithi187 conmon[36358]: cluster 2021-11-20T05:34:25.224186+0000 mon.smithi058 (mon.0) 526 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi058.rxapzd=up:active,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:34:25.712 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2021-11-20T05:34:25.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:25.713 DEBUG:teuthology.orchestra.run.smithi058:> (cd / && exec bash -c ' 2021-11-20T05:34:25.714 DEBUG:teuthology.orchestra.run.smithi058:> set -e 2021-11-20T05:34:25.714 DEBUG:teuthology.orchestra.run.smithi058:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2021-11-20T05:34:25.714 DEBUG:teuthology.orchestra.run.smithi058:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2021-11-20T05:34:25.715 DEBUG:teuthology.orchestra.run.smithi058:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2021-11-20T05:34:25.715 DEBUG:teuthology.orchestra.run.smithi058:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2021-11-20T05:34:25.715 DEBUG:teuthology.orchestra.run.smithi058:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2021-11-20T05:34:25.716 DEBUG:teuthology.orchestra.run.smithi058:> ') 2021-11-20T05:34:25.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:25.950 DEBUG:teuthology.orchestra.run.smithi058:> (cd / && exec bash -c ' 2021-11-20T05:34:25.951 DEBUG:teuthology.orchestra.run.smithi058:> set -e 2021-11-20T05:34:25.951 DEBUG:teuthology.orchestra.run.smithi058:> sudo ip link set brx.0 up 2021-11-20T05:34:25.951 DEBUG:teuthology.orchestra.run.smithi058:> sudo ip link set dev brx.0 master ceph-brx 2021-11-20T05:34:25.952 DEBUG:teuthology.orchestra.run.smithi058:> ') 2021-11-20T05:34:26.081 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-11-20T05:34:26.221 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.219+0000 7f4ed0b4e700 1 -- 172.21.15.58:0/3373738382 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4ecc0f9720 msgr2=0x7f4ecc0fbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:26.222 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.219+0000 7f4ed0b4e700 1 --2- 172.21.15.58:0/3373738382 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4ecc0f9720 0x7f4ecc0fbb10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f4ec0004660 tx=0x7f4ec0009b30).stop 2021-11-20T05:34:26.222 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.219+0000 7f4ed0b4e700 1 -- 172.21.15.58:0/3373738382 shutdown_connections 2021-11-20T05:34:26.222 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.219+0000 7f4ed0b4e700 1 --2- 172.21.15.58:0/3373738382 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4ecc0f9720 0x7f4ecc0fbb10 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:26.223 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.219+0000 7f4ed0b4e700 1 --2- 172.21.15.58:0/3373738382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ecc0600e0 0x7f4ecc0f91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:26.223 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.219+0000 7f4ed0b4e700 1 -- 172.21.15.58:0/3373738382 >> 172.21.15.58:0/3373738382 conn(0x7f4ecc0f0ba0 msgr2=0x7f4ecc0f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:26.223 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.219+0000 7f4ed0b4e700 1 -- 172.21.15.58:0/3373738382 shutdown_connections 2021-11-20T05:34:26.224 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.219+0000 7f4ed0b4e700 1 -- 172.21.15.58:0/3373738382 wait complete. 2021-11-20T05:34:26.224 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.219+0000 7f4ed0b4e700 1 Processor -- start 2021-11-20T05:34:26.224 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.220+0000 7f4ed0b4e700 1 -- start start 2021-11-20T05:34:26.224 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.220+0000 7f4ed0b4e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ecc0600e0 0x7f4ecc106d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:26.224 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.220+0000 7f4ed0b4e700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4ecc0f9720 0x7f4ecc107240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:26.225 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.220+0000 7f4ed0b4e700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ecc1077f0 con 0x7f4ecc0f9720 2021-11-20T05:34:26.225 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.220+0000 7f4ed0b4e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ecc107ff0 con 0x7f4ecc0600e0 2021-11-20T05:34:26.225 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.220+0000 7f4ecaffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4ecc0f9720 0x7f4ecc107240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:26.226 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.220+0000 7f4ecaffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4ecc0f9720 0x7f4ecc107240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40638/0 (socket says 172.21.15.58:40638) 2021-11-20T05:34:26.227 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.220+0000 7f4ecaffd700 1 -- 172.21.15.58:0/140866815 learned_addr learned my addr 172.21.15.58:0/140866815 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:26.227 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.220+0000 7f4ecb7fe700 1 --2- 172.21.15.58:0/140866815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ecc0600e0 0x7f4ecc106d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:26.227 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.220+0000 7f4ecaffd700 1 -- 172.21.15.58:0/140866815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ecc0600e0 msgr2=0x7f4ecc106d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:26.227 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.220+0000 7f4ecaffd700 1 --2- 172.21.15.58:0/140866815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ecc0600e0 0x7f4ecc106d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:26.228 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.220+0000 7f4ecaffd700 1 -- 172.21.15.58:0/140866815 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec0005040 con 0x7f4ecc0f9720 2021-11-20T05:34:26.228 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.221+0000 7f4ecaffd700 1 --2- 172.21.15.58:0/140866815 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4ecc0f9720 0x7f4ecc107240 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f4ec0004e30 tx=0x7f4ec0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:26.228 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.221+0000 7f4eb3fff700 1 -- 172.21.15.58:0/140866815 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ec0018070 con 0x7f4ecc0f9720 2021-11-20T05:34:26.228 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.221+0000 7f4eb3fff700 1 -- 172.21.15.58:0/140866815 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ec0013410 con 0x7f4ecc0f9720 2021-11-20T05:34:26.228 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.221+0000 7f4ed0b4e700 1 -- 172.21.15.58:0/140866815 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ecc108270 con 0x7f4ecc0f9720 2021-11-20T05:34:26.229 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.221+0000 7f4eb3fff700 1 -- 172.21.15.58:0/140866815 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ec001de20 con 0x7f4ecc0f9720 2021-11-20T05:34:26.229 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.221+0000 7f4ed0b4e700 1 -- 172.21.15.58:0/140866815 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ecc1087d0 con 0x7f4ecc0f9720 2021-11-20T05:34:26.230 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.222+0000 7f4ed0b4e700 1 -- 172.21.15.58:0/140866815 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ecc0f4850 con 0x7f4ecc0f9720 2021-11-20T05:34:26.230 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.222+0000 7f4eb3fff700 1 -- 172.21.15.58:0/140866815 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4ec001c450 con 0x7f4ecc0f9720 2021-11-20T05:34:26.231 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.222+0000 7f4eb3fff700 1 --2- 172.21.15.58:0/140866815 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4eb405c300 0x7f4eb405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:26.231 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.222+0000 7f4eb3fff700 1 -- 172.21.15.58:0/140866815 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f4ec0077a30 con 0x7f4ecc0f9720 2021-11-20T05:34:26.231 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.223+0000 7f4ecb7fe700 1 --2- 172.21.15.58:0/140866815 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4eb405c300 0x7f4eb405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:26.232 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.223+0000 7f4ecb7fe700 1 --2- 172.21.15.58:0/140866815 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4eb405c300 0x7f4eb405e7c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f4ebc004ce0 tx=0x7f4ebc00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:26.232 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.226+0000 7f4eb3fff700 1 -- 172.21.15.58:0/140866815 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ec004a770 con 0x7f4ecc0f9720 2021-11-20T05:34:26.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.386+0000 7f4ed0b4e700 1 -- 172.21.15.58:0/140866815 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ecc0fecf0 con 0x7f4ecc0f9720 2021-11-20T05:34:26.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.387+0000 7f4eb3fff700 1 -- 172.21.15.58:0/140866815 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+14084 (secure 0 0 0) 0x7f4ec004dd90 con 0x7f4ecc0f9720 2021-11-20T05:34:26.390 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:34:26.390 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":42,"fsid":"b33f8c4a-49c2-11ec-8c2c-001a4aab830c","created":"2021-11-20T05:28:54.787342+0000","modified":"2021-11-20T05:34:08.318974+0000","last_up_change":"2021-11-20T05:33:10.122001+0000","last_in_change":"2021-11-20T05:32:53.577246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T05:32:27.068478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2021-11-20T05:34:04.784086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2021-11-20T05:34:05.738224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e2f5103a-e0d2-4bc3-88e1-f28420c98071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6802","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6803","nonce":2375873844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6804","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6805","nonce":2375873844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6808","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6809","nonce":2375873844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6806","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6807","nonce":2375873844}]},"public_addr":"172.21.15.58:6803/2375873844","cluster_addr":"172.21.15.58:6805/2375873844","heartbeat_back_addr":"172.21.15.58:6809/2375873844","heartbeat_front_addr":"172.21.15.58:6807/2375873844","state":["exists","up"]},{"osd":1,"uuid":"e0243e40-c76e-4c68-8ac0-8f555035e948","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6810","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6811","nonce":3875723869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6812","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6813","nonce":3875723869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6816","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6817","nonce":3875723869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6814","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6815","nonce":3875723869}]},"public_addr":"172.21.15.58:6811/3875723869","cluster_addr":"172.21.15.58:6813/3875723869","heartbeat_back_addr":"172.21.15.58:6817/3875723869","heartbeat_front_addr":"172.21.15.58:6815/3875723869","state":["exists","up"]},{"osd":2,"uuid":"9b35ec54-f60f-49c5-ab1b-d2c5b5eb72e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6818","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6819","nonce":745015076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6820","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6821","nonce":745015076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6824","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6825","nonce":745015076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6822","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6823","nonce":745015076}]},"public_addr":"172.21.15.58:6819/745015076","cluster_addr":"172.21.15.58:6821/745015076","heartbeat_back_addr":"172.21.15.58:6825/745015076","heartbeat_front_addr":"172.21.15.58:6823/745015076","state":["exists","up"]},{"osd":3,"uuid":"8390f146-b7e1-4039-83d3-7ed1c832dc8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6801","nonce":1941886381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6803","nonce":1941886381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6807","nonce":1941886381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6805","nonce":1941886381}]},"public_addr":"172.21.15.187:6801/1941886381","cluster_addr":"172.21.15.187:6803/1941886381","heartbeat_back_addr":"172.21.15.187:6807/1941886381","heartbeat_front_addr":"172.21.15.187:6805/1941886381","state":["exists","up"]},{"osd":4,"uuid":"89dc0c2f-c52c-48ca-b3ab-792e299d494d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6809","nonce":3483552299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6811","nonce":3483552299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6815","nonce":3483552299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6813","nonce":3483552299}]},"public_addr":"172.21.15.187:6809/3483552299","cluster_addr":"172.21.15.187:6811/3483552299","heartbeat_back_addr":"172.21.15.187:6815/3483552299","heartbeat_front_addr":"172.21.15.187:6813/3483552299","state":["exists","up"]},{"osd":5,"uuid":"8190ded3-7b78-4d71-9c23-e5bc4112fecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6817","nonce":566077514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6819","nonce":566077514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6823","nonce":566077514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6821","nonce":566077514}]},"public_addr":"172.21.15.187:6817/566077514","cluster_addr":"172.21.15.187:6819/566077514","heartbeat_back_addr":"172.21.15.187:6823/566077514","heartbeat_front_addr":"172.21.15.187:6821/566077514","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:31:38.522095+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:31:56.685873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:15.227272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:31.493769+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:50.070298+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:33:08.140441+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.58:6801/2422532259":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/1616356583":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/2030112153":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/3301163446":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6801/1261811851":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6800/1261811851":"2021-11-21T05:29:42.633679+0000","172.21.15.58:0/1447224424":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6800/324085383":"2021-11-21T05:29:22.266837+0000","172.21.15.58:6801/324085383":"2021-11-21T05:29:22.266837+0000","172.21.15.58:0/574129054":"2021-11-21T05:29:22.266837+0000","172.21.15.58:6800/2422532259":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/2571399354":"2021-11-21T05:29:22.266837+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T05:34:26.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.389+0000 7f4ed0b4e700 1 -- 172.21.15.58:0/140866815 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4eb405c300 msgr2=0x7f4eb405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:26.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.389+0000 7f4ed0b4e700 1 --2- 172.21.15.58:0/140866815 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4eb405c300 0x7f4eb405e7c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f4ebc004ce0 tx=0x7f4ebc00b040).stop 2021-11-20T05:34:26.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.389+0000 7f4ed0b4e700 1 -- 172.21.15.58:0/140866815 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4ecc0f9720 msgr2=0x7f4ecc107240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:26.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.389+0000 7f4ed0b4e700 1 --2- 172.21.15.58:0/140866815 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4ecc0f9720 0x7f4ecc107240 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f4ec0004e30 tx=0x7f4ec0009b30).stop 2021-11-20T05:34:26.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.389+0000 7f4ed0b4e700 1 -- 172.21.15.58:0/140866815 shutdown_connections 2021-11-20T05:34:26.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.389+0000 7f4ed0b4e700 1 --2- 172.21.15.58:0/140866815 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4eb405c300 0x7f4eb405e7c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:26.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.389+0000 7f4ed0b4e700 1 --2- 172.21.15.58:0/140866815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ecc0600e0 0x7f4ecc106d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:26.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.389+0000 7f4ed0b4e700 1 --2- 172.21.15.58:0/140866815 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4ecc0f9720 0x7f4ecc107240 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:26.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.389+0000 7f4ed0b4e700 1 -- 172.21.15.58:0/140866815 >> 172.21.15.58:0/140866815 conn(0x7f4ecc0f0ba0 msgr2=0x7f4ecc0f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:26.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.389+0000 7f4ed0b4e700 1 -- 172.21.15.58:0/140866815 shutdown_connections 2021-11-20T05:34:26.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.389+0000 7f4ed0b4e700 1 -- 172.21.15.58:0/140866815 wait complete. 2021-11-20T05:34:26.401 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-20T05:32:27.068478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2021-11-20T05:34:04.784086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2021-11-20T05:34:05.738224+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-11-20T05:34:26.402 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2021-11-20T05:34:26.530 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.528+0000 7f17ae323700 1 -- 172.21.15.58:0/3581109351 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f17a80f7920 msgr2=0x7f17a80f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:26.531 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.528+0000 7f17ae323700 1 --2- 172.21.15.58:0/3581109351 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f17a80f7920 0x7f17a80f7d80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f1790004660 tx=0x7f1790009b30).stop 2021-11-20T05:34:26.531 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.528+0000 7f17ae323700 1 -- 172.21.15.58:0/3581109351 shutdown_connections 2021-11-20T05:34:26.531 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.528+0000 7f17ae323700 1 --2- 172.21.15.58:0/3581109351 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f17a80f7920 0x7f17a80f7d80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:26.531 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.528+0000 7f17ae323700 1 --2- 172.21.15.58:0/3581109351 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17a80f6680 0x7f17a80f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:26.532 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.528+0000 7f17ae323700 1 -- 172.21.15.58:0/3581109351 >> 172.21.15.58:0/3581109351 conn(0x7f17a80f1de0 msgr2=0x7f17a80f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:26.532 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.528+0000 7f17ae323700 1 -- 172.21.15.58:0/3581109351 shutdown_connections 2021-11-20T05:34:26.532 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.528+0000 7f17ae323700 1 -- 172.21.15.58:0/3581109351 wait complete. 2021-11-20T05:34:26.533 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.528+0000 7f17ae323700 1 Processor -- start 2021-11-20T05:34:26.533 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.529+0000 7f17ae323700 1 -- start start 2021-11-20T05:34:26.533 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.529+0000 7f17ae323700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f17a80f6680 0x7f17a806b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:26.533 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.529+0000 7f17ae323700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17a80f7920 0x7f17a8069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:26.534 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.529+0000 7f17ae323700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a806bd70 con 0x7f17a80f6680 2021-11-20T05:34:26.534 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.529+0000 7f17ae323700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a806a310 con 0x7f17a80f7920 2021-11-20T05:34:26.534 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.529+0000 7f17a7fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f17a80f6680 0x7f17a806b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:26.534 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.529+0000 7f17a7fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f17a80f6680 0x7f17a806b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40646/0 (socket says 172.21.15.58:40646) 2021-11-20T05:34:26.535 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.529+0000 7f17a7fff700 1 -- 172.21.15.58:0/3578443907 learned_addr learned my addr 172.21.15.58:0/3578443907 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:26.535 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.529+0000 7f17a77fe700 1 --2- 172.21.15.58:0/3578443907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17a80f7920 0x7f17a8069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:26.535 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.529+0000 7f17a7fff700 1 -- 172.21.15.58:0/3578443907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17a80f7920 msgr2=0x7f17a8069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:26.535 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.529+0000 7f17a7fff700 1 --2- 172.21.15.58:0/3578443907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17a80f7920 0x7f17a8069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:26.536 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.529+0000 7f17a7fff700 1 -- 172.21.15.58:0/3578443907 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1790005040 con 0x7f17a80f6680 2021-11-20T05:34:26.536 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.529+0000 7f17a7fff700 1 --2- 172.21.15.58:0/3578443907 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f17a80f6680 0x7f17a806b7c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f179800e7c0 tx=0x7f179800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:26.536 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.530+0000 7f17a4ff9700 1 -- 172.21.15.58:0/3578443907 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17980092e0 con 0x7f17a80f6680 2021-11-20T05:34:26.536 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.530+0000 7f17ae323700 1 -- 172.21.15.58:0/3578443907 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a806a590 con 0x7f17a80f6680 2021-11-20T05:34:26.537 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.530+0000 7f17a4ff9700 1 -- 172.21.15.58:0/3578443907 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f179800f040 con 0x7f17a80f6680 2021-11-20T05:34:26.537 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.530+0000 7f17a4ff9700 1 -- 172.21.15.58:0/3578443907 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17980147c0 con 0x7f17a80f6680 2021-11-20T05:34:26.537 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.530+0000 7f17ae323700 1 -- 172.21.15.58:0/3578443907 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a806ab50 con 0x7f17a80f6680 2021-11-20T05:34:26.537 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.530+0000 7f17ae323700 1 -- 172.21.15.58:0/3578443907 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a8027a00 con 0x7f17a80f6680 2021-11-20T05:34:26.538 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.531+0000 7f17a4ff9700 1 -- 172.21.15.58:0/3578443907 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f179801b030 con 0x7f17a80f6680 2021-11-20T05:34:26.539 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.531+0000 7f17a4ff9700 1 --2- 172.21.15.58:0/3578443907 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f179405c2a0 0x7f179405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:26.539 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.531+0000 7f17a4ff9700 1 -- 172.21.15.58:0/3578443907 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f1798075e80 con 0x7f17a80f6680 2021-11-20T05:34:26.540 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.531+0000 7f17a77fe700 1 --2- 172.21.15.58:0/3578443907 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f179405c2a0 0x7f179405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:26.540 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.532+0000 7f17a77fe700 1 --2- 172.21.15.58:0/3578443907 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f179405c2a0 0x7f179405e760 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f1790005010 tx=0x7f1790016040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:26.540 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.535+0000 7f17a4ff9700 1 -- 172.21.15.58:0/3578443907 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1798048b40 con 0x7f17a80f6680 2021-11-20T05:34:26.695 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.696+0000 7f17ae323700 1 -- 172.21.15.58:0/3578443907 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f17a80fee10 con 0x7f17a80f6680 2021-11-20T05:34:26.696 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.696+0000 7f17a4ff9700 1 -- 172.21.15.58:0/3578443907 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v42) v1 ==== 110+0+10 (secure 0 0 0) 0x7f1798019020 con 0x7f17a80f6680 2021-11-20T05:34:26.696 INFO:teuthology.orchestra.run.smithi058.stdout:pg_num: 1 2021-11-20T05:34:26.699 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.698+0000 7f17ae323700 1 -- 172.21.15.58:0/3578443907 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f179405c2a0 msgr2=0x7f179405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:26.699 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.698+0000 7f17ae323700 1 --2- 172.21.15.58:0/3578443907 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f179405c2a0 0x7f179405e760 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f1790005010 tx=0x7f1790016040).stop 2021-11-20T05:34:26.699 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.698+0000 7f17ae323700 1 -- 172.21.15.58:0/3578443907 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f17a80f6680 msgr2=0x7f17a806b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:26.700 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.698+0000 7f17ae323700 1 --2- 172.21.15.58:0/3578443907 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f17a80f6680 0x7f17a806b7c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f179800e7c0 tx=0x7f179800b040).stop 2021-11-20T05:34:26.700 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.699+0000 7f17ae323700 1 -- 172.21.15.58:0/3578443907 shutdown_connections 2021-11-20T05:34:26.700 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.699+0000 7f17ae323700 1 --2- 172.21.15.58:0/3578443907 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f179405c2a0 0x7f179405e760 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:26.701 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.699+0000 7f17ae323700 1 --2- 172.21.15.58:0/3578443907 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f17a80f6680 0x7f17a806b7c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:26.701 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.699+0000 7f17ae323700 1 --2- 172.21.15.58:0/3578443907 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f17a80f7920 0x7f17a8069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:26.701 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.699+0000 7f17ae323700 1 -- 172.21.15.58:0/3578443907 >> 172.21.15.58:0/3578443907 conn(0x7f17a80f1de0 msgr2=0x7f17a80fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:26.702 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.699+0000 7f17ae323700 1 -- 172.21.15.58:0/3578443907 shutdown_connections 2021-11-20T05:34:26.702 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.699+0000 7f17ae323700 1 -- 172.21.15.58:0/3578443907 wait complete. 2021-11-20T05:34:26.712 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2021-11-20T05:34:26.847 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.847+0000 7fbd8b59e700 1 -- 172.21.15.58:0/2776690160 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd8c0f47a0 msgr2=0x7fbd8c0f4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:26.848 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.847+0000 7fbd8b59e700 1 --2- 172.21.15.58:0/2776690160 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd8c0f47a0 0x7fbd8c0f4bc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fbd7c004660 tx=0x7fbd7c009b30).stop 2021-11-20T05:34:26.848 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.847+0000 7fbd8b59e700 1 -- 172.21.15.58:0/2776690160 shutdown_connections 2021-11-20T05:34:26.848 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.847+0000 7fbd8b59e700 1 --2- 172.21.15.58:0/2776690160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8c0f5140 0x7fbd8c0f78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:26.848 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.847+0000 7fbd8b59e700 1 --2- 172.21.15.58:0/2776690160 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd8c0f47a0 0x7fbd8c0f4bc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:26.849 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.847+0000 7fbd8b59e700 1 -- 172.21.15.58:0/2776690160 >> 172.21.15.58:0/2776690160 conn(0x7fbd8c0f02c0 msgr2=0x7fbd8c0f26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:26.849 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.847+0000 7fbd8b59e700 1 -- 172.21.15.58:0/2776690160 shutdown_connections 2021-11-20T05:34:26.849 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.847+0000 7fbd8b59e700 1 -- 172.21.15.58:0/2776690160 wait complete. 2021-11-20T05:34:26.857 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.848+0000 7fbd8b59e700 1 Processor -- start 2021-11-20T05:34:26.858 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.848+0000 7fbd8b59e700 1 -- start start 2021-11-20T05:34:26.858 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.848+0000 7fbd8b59e700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd8c0f47a0 0x7fbd8c106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:26.858 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.848+0000 7fbd8b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8c0f5140 0x7fbd8c1073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:26.858 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.848+0000 7fbd8b59e700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8c107950 con 0x7fbd8c0f47a0 2021-11-20T05:34:26.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.848+0000 7fbd8b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8c108150 con 0x7fbd8c0f5140 2021-11-20T05:34:26.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.848+0000 7fbd8a59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd8c0f47a0 0x7fbd8c106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:26.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.848+0000 7fbd8a59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd8c0f47a0 0x7fbd8c106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40656/0 (socket says 172.21.15.58:40656) 2021-11-20T05:34:26.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.848+0000 7fbd8a59c700 1 -- 172.21.15.58:0/3636422352 learned_addr learned my addr 172.21.15.58:0/3636422352 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:26.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.848+0000 7fbd89d9b700 1 --2- 172.21.15.58:0/3636422352 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8c0f5140 0x7fbd8c1073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:26.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.848+0000 7fbd8a59c700 1 -- 172.21.15.58:0/3636422352 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8c0f5140 msgr2=0x7fbd8c1073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:26.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.848+0000 7fbd8a59c700 1 --2- 172.21.15.58:0/3636422352 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8c0f5140 0x7fbd8c1073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:26.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.848+0000 7fbd8a59c700 1 -- 172.21.15.58:0/3636422352 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7c005040 con 0x7fbd8c0f47a0 2021-11-20T05:34:26.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.848+0000 7fbd89d9b700 1 --2- 172.21.15.58:0/3636422352 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8c0f5140 0x7fbd8c1073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:34:26.861 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.848+0000 7fbd8a59c700 1 --2- 172.21.15.58:0/3636422352 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd8c0f47a0 0x7fbd8c106eb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fbd7c004e30 tx=0x7fbd7c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:26.861 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.849+0000 7fbd7affd700 1 -- 172.21.15.58:0/3636422352 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd7c018070 con 0x7fbd8c0f47a0 2021-11-20T05:34:26.861 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.849+0000 7fbd7affd700 1 -- 172.21.15.58:0/3636422352 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd7c013410 con 0x7fbd8c0f47a0 2021-11-20T05:34:26.861 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.849+0000 7fbd8b59e700 1 -- 172.21.15.58:0/3636422352 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd8c1083d0 con 0x7fbd8c0f47a0 2021-11-20T05:34:26.861 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.849+0000 7fbd7affd700 1 -- 172.21.15.58:0/3636422352 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd7c01c3c0 con 0x7fbd8c0f47a0 2021-11-20T05:34:26.862 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.849+0000 7fbd8b59e700 1 -- 172.21.15.58:0/3636422352 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd8c108930 con 0x7fbd8c0f47a0 2021-11-20T05:34:26.862 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.850+0000 7fbd8b59e700 1 -- 172.21.15.58:0/3636422352 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd8c0f3f50 con 0x7fbd8c0f47a0 2021-11-20T05:34:26.862 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.850+0000 7fbd7affd700 1 -- 172.21.15.58:0/3636422352 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbd7c01d420 con 0x7fbd8c0f47a0 2021-11-20T05:34:26.862 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.850+0000 7fbd7affd700 1 --2- 172.21.15.58:0/3636422352 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbd7405c2b0 0x7fbd7405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:26.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.850+0000 7fbd7affd700 1 -- 172.21.15.58:0/3636422352 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fbd7c077a90 con 0x7fbd8c0f47a0 2021-11-20T05:34:26.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.850+0000 7fbd89d9b700 1 --2- 172.21.15.58:0/3636422352 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbd7405c2b0 0x7fbd7405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:26.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.851+0000 7fbd89d9b700 1 --2- 172.21.15.58:0/3636422352 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbd7405c2b0 0x7fbd7405e770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fbd8c0f5580 tx=0x7fbd80009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:26.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:26.854+0000 7fbd7affd700 1 -- 172.21.15.58:0/3636422352 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd7c04a7d0 con 0x7fbd8c0f47a0 2021-11-20T05:34:27.016 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.016+0000 7fbd8b59e700 1 -- 172.21.15.58:0/3636422352 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fbd8c0fecb0 con 0x7fbd8c0f47a0 2021-11-20T05:34:27.018 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.016+0000 7fbd7affd700 1 -- 172.21.15.58:0/3636422352 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v42) v1 ==== 107+0+11 (secure 0 0 0) 0x7fbd7c04ddf0 con 0x7fbd8c0f47a0 2021-11-20T05:34:27.019 INFO:teuthology.orchestra.run.smithi058.stdout:pg_num: 32 2021-11-20T05:34:27.020 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.018+0000 7fbd8b59e700 1 -- 172.21.15.58:0/3636422352 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbd7405c2b0 msgr2=0x7fbd7405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.020 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.018+0000 7fbd8b59e700 1 --2- 172.21.15.58:0/3636422352 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbd7405c2b0 0x7fbd7405e770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fbd8c0f5580 tx=0x7fbd80009be0).stop 2021-11-20T05:34:27.021 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.018+0000 7fbd8b59e700 1 -- 172.21.15.58:0/3636422352 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd8c0f47a0 msgr2=0x7fbd8c106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.021 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.018+0000 7fbd8b59e700 1 --2- 172.21.15.58:0/3636422352 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd8c0f47a0 0x7fbd8c106eb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fbd7c004e30 tx=0x7fbd7c009b30).stop 2021-11-20T05:34:27.022 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.018+0000 7fbd8b59e700 1 -- 172.21.15.58:0/3636422352 shutdown_connections 2021-11-20T05:34:27.022 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.018+0000 7fbd8b59e700 1 --2- 172.21.15.58:0/3636422352 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbd7405c2b0 0x7fbd7405e770 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.022 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.018+0000 7fbd8b59e700 1 --2- 172.21.15.58:0/3636422352 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd8c0f47a0 0x7fbd8c106eb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.022 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.018+0000 7fbd8b59e700 1 --2- 172.21.15.58:0/3636422352 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8c0f5140 0x7fbd8c1073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.018+0000 7fbd8b59e700 1 -- 172.21.15.58:0/3636422352 >> 172.21.15.58:0/3636422352 conn(0x7fbd8c0f02c0 msgr2=0x7fbd8c0fb310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:27.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.018+0000 7fbd8b59e700 1 -- 172.21.15.58:0/3636422352 shutdown_connections 2021-11-20T05:34:27.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.018+0000 7fbd8b59e700 1 -- 172.21.15.58:0/3636422352 wait complete. 2021-11-20T05:34:27.032 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2021-11-20T05:34:27.164 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.157+0000 7efc8304b700 1 -- 172.21.15.58:0/3061922332 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efc7c0f5590 msgr2=0x7efc7c0f59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.164 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.157+0000 7efc8304b700 1 --2- 172.21.15.58:0/3061922332 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efc7c0f5590 0x7efc7c0f59b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7efc740099e0 tx=0x7efc74004d10).stop 2021-11-20T05:34:27.164 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.157+0000 7efc8304b700 1 -- 172.21.15.58:0/3061922332 shutdown_connections 2021-11-20T05:34:27.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.157+0000 7efc8304b700 1 --2- 172.21.15.58:0/3061922332 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc7c0f6830 0x7efc7c0f6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.157+0000 7efc8304b700 1 --2- 172.21.15.58:0/3061922332 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efc7c0f5590 0x7efc7c0f59b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.157+0000 7efc8304b700 1 -- 172.21.15.58:0/3061922332 >> 172.21.15.58:0/3061922332 conn(0x7efc7c0f0d10 msgr2=0x7efc7c0f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:27.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.157+0000 7efc8304b700 1 -- 172.21.15.58:0/3061922332 shutdown_connections 2021-11-20T05:34:27.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.157+0000 7efc8304b700 1 -- 172.21.15.58:0/3061922332 wait complete. 2021-11-20T05:34:27.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.157+0000 7efc8304b700 1 Processor -- start 2021-11-20T05:34:27.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.157+0000 7efc8304b700 1 -- start start 2021-11-20T05:34:27.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.158+0000 7efc8304b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efc7c0f5590 0x7efc7c104c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.158+0000 7efc8304b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc7c0f6830 0x7efc7c105120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.158+0000 7efc8304b700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c105720 con 0x7efc7c0f5590 2021-11-20T05:34:27.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.158+0000 7efc8304b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c105860 con 0x7efc7c0f6830 2021-11-20T05:34:27.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.158+0000 7efc80de7700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efc7c0f5590 0x7efc7c104c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.158+0000 7efc80de7700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efc7c0f5590 0x7efc7c104c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40666/0 (socket says 172.21.15.58:40666) 2021-11-20T05:34:27.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.158+0000 7efc80de7700 1 -- 172.21.15.58:0/348289470 learned_addr learned my addr 172.21.15.58:0/348289470 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:27.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.158+0000 7efc7bfff700 1 --2- 172.21.15.58:0/348289470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc7c0f6830 0x7efc7c105120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.158+0000 7efc80de7700 1 -- 172.21.15.58:0/348289470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc7c0f6830 msgr2=0x7efc7c105120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.158+0000 7efc80de7700 1 --2- 172.21.15.58:0/348289470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc7c0f6830 0x7efc7c105120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.158+0000 7efc80de7700 1 -- 172.21.15.58:0/348289470 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc74009640 con 0x7efc7c0f5590 2021-11-20T05:34:27.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.158+0000 7efc7bfff700 1 --2- 172.21.15.58:0/348289470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc7c0f6830 0x7efc7c105120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:34:27.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.158+0000 7efc80de7700 1 --2- 172.21.15.58:0/348289470 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efc7c0f5590 0x7efc7c104c30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7efc74004750 tx=0x7efc74004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:27.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.158+0000 7efc797fa700 1 -- 172.21.15.58:0/348289470 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc74007800 con 0x7efc7c0f5590 2021-11-20T05:34:27.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.159+0000 7efc797fa700 1 -- 172.21.15.58:0/348289470 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc74007960 con 0x7efc7c0f5590 2021-11-20T05:34:27.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.159+0000 7efc8304b700 1 -- 172.21.15.58:0/348289470 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7c10a210 con 0x7efc7c0f5590 2021-11-20T05:34:27.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.159+0000 7efc797fa700 1 -- 172.21.15.58:0/348289470 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc7401b750 con 0x7efc7c0f5590 2021-11-20T05:34:27.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.159+0000 7efc8304b700 1 -- 172.21.15.58:0/348289470 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7c10a770 con 0x7efc7c0f5590 2021-11-20T05:34:27.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.160+0000 7efc8304b700 1 -- 172.21.15.58:0/348289470 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7c027a00 con 0x7efc7c0f5590 2021-11-20T05:34:27.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.160+0000 7efc797fa700 1 -- 172.21.15.58:0/348289470 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7efc7402b070 con 0x7efc7c0f5590 2021-11-20T05:34:27.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.160+0000 7efc797fa700 1 --2- 172.21.15.58:0/348289470 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efc6405c2a0 0x7efc6405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.160+0000 7efc797fa700 1 -- 172.21.15.58:0/348289470 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7efc7407a750 con 0x7efc7c0f5590 2021-11-20T05:34:27.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.161+0000 7efc7bfff700 1 --2- 172.21.15.58:0/348289470 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efc6405c2a0 0x7efc6405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.161+0000 7efc7bfff700 1 --2- 172.21.15.58:0/348289470 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efc6405c2a0 0x7efc6405e760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7efc70007fc0 tx=0x7efc7000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:27.172 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.164+0000 7efc797fa700 1 -- 172.21.15.58:0/348289470 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc7404d490 con 0x7efc7c0f5590 2021-11-20T05:34:27.329 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.329+0000 7efc8304b700 1 -- 172.21.15.58:0/348289470 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7efc7c0fcc10 con 0x7efc7c0f5590 2021-11-20T05:34:27.329 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.329+0000 7efc797fa700 1 -- 172.21.15.58:0/348289470 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v42) v1 ==== 107+0+11 (secure 0 0 0) 0x7efc7401f0b0 con 0x7efc7c0f5590 2021-11-20T05:34:27.331 INFO:teuthology.orchestra.run.smithi058.stdout:pg_num: 32 2021-11-20T05:34:27.332 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.331+0000 7efc8304b700 1 -- 172.21.15.58:0/348289470 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efc6405c2a0 msgr2=0x7efc6405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.332 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.331+0000 7efc8304b700 1 --2- 172.21.15.58:0/348289470 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efc6405c2a0 0x7efc6405e760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7efc70007fc0 tx=0x7efc7000b040).stop 2021-11-20T05:34:27.332 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.331+0000 7efc8304b700 1 -- 172.21.15.58:0/348289470 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efc7c0f5590 msgr2=0x7efc7c104c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.332 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.331+0000 7efc8304b700 1 --2- 172.21.15.58:0/348289470 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efc7c0f5590 0x7efc7c104c30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7efc74004750 tx=0x7efc74004d10).stop 2021-11-20T05:34:27.333 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.331+0000 7efc8304b700 1 -- 172.21.15.58:0/348289470 shutdown_connections 2021-11-20T05:34:27.333 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.331+0000 7efc8304b700 1 --2- 172.21.15.58:0/348289470 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efc6405c2a0 0x7efc6405e760 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.333 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.331+0000 7efc8304b700 1 --2- 172.21.15.58:0/348289470 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efc7c0f5590 0x7efc7c104c30 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.333 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.331+0000 7efc8304b700 1 --2- 172.21.15.58:0/348289470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc7c0f6830 0x7efc7c105120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.334 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.331+0000 7efc8304b700 1 -- 172.21.15.58:0/348289470 >> 172.21.15.58:0/348289470 conn(0x7efc7c0f0d10 msgr2=0x7efc7c0fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:27.334 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.331+0000 7efc8304b700 1 -- 172.21.15.58:0/348289470 shutdown_connections 2021-11-20T05:34:27.334 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.331+0000 7efc8304b700 1 -- 172.21.15.58:0/348289470 wait complete. 2021-11-20T05:34:27.342 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:27 smithi058 conmon[33537]: audit 2021-11-20T05:34:26.209430+0000 mon.smithi058 (mon.0) 527 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:27.343 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:27 smithi058 conmon[33537]: cluster 2021-11-20T05:34:26.213073+0000 mgr.smithi058.zagdlk (mgr.14164) 202 : cluster [DBG] pgmap v164: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s wr, 6 op/s 2021-11-20T05:34:27.343 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:27 smithi058 conmon[33537]: audit 2021-11-20T05:34:26.388019+0000 mon.smithi058 (mon.0) 528 : audit [DBG] from='client.? 172.21.15.58:0/140866815' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T05:34:27.343 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:27 smithi058 conmon[33537]: audit 2021-11-20T05:34:26.697516+0000 mon.smithi058 (mon.0) 529 : audit [DBG] from='client.? 172.21.15.58:0/3578443907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T05:34:27.344 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:27 smithi058 conmon[33537]: audit 2021-11-20T05:34:27.017551+0000 mon.smithi058 (mon.0) 530 : audit [DBG] from='client.? 172.21.15.58:0/3636422352' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T05:34:27.345 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-11-20T05:34:27.345 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:34:27.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.479+0000 7fcb06bc2700 1 -- 172.21.15.58:0/2273715149 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcb000f5460 msgr2=0x7fcb000f5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.484 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.479+0000 7fcb06bc2700 1 --2- 172.21.15.58:0/2273715149 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcb000f5460 0x7fcb000f5880 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fcafc004660 tx=0x7fcafc009b30).stop 2021-11-20T05:34:27.484 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.479+0000 7fcb06bc2700 1 -- 172.21.15.58:0/2273715149 shutdown_connections 2021-11-20T05:34:27.484 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.479+0000 7fcb06bc2700 1 --2- 172.21.15.58:0/2273715149 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb000f6700 0x7fcb000f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.485 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.479+0000 7fcb06bc2700 1 --2- 172.21.15.58:0/2273715149 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcb000f5460 0x7fcb000f5880 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.485 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.479+0000 7fcb06bc2700 1 -- 172.21.15.58:0/2273715149 >> 172.21.15.58:0/2273715149 conn(0x7fcb000f0ba0 msgr2=0x7fcb000f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:27.485 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.479+0000 7fcb06bc2700 1 -- 172.21.15.58:0/2273715149 shutdown_connections 2021-11-20T05:34:27.485 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.480+0000 7fcb06bc2700 1 -- 172.21.15.58:0/2273715149 wait complete. 2021-11-20T05:34:27.485 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.480+0000 7fcb06bc2700 1 Processor -- start 2021-11-20T05:34:27.486 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.480+0000 7fcb06bc2700 1 -- start start 2021-11-20T05:34:27.486 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.480+0000 7fcb06bc2700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcb000f5460 0x7fcb00109a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.486 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.480+0000 7fcb06bc2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb000f6700 0x7fcb0010bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.486 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.480+0000 7fcb06bc2700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb000601f0 con 0x7fcb000f5460 2021-11-20T05:34:27.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.480+0000 7fcb06bc2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb00060330 con 0x7fcb000f6700 2021-11-20T05:34:27.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.480+0000 7fcb05bc0700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcb000f5460 0x7fcb00109a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.480+0000 7fcb05bc0700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcb000f5460 0x7fcb00109a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40676/0 (socket says 172.21.15.58:40676) 2021-11-20T05:34:27.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.480+0000 7fcb05bc0700 1 -- 172.21.15.58:0/346600577 learned_addr learned my addr 172.21.15.58:0/346600577 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:27.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.480+0000 7fcb053bf700 1 --2- 172.21.15.58:0/346600577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb000f6700 0x7fcb0010bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.488 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.481+0000 7fcb05bc0700 1 -- 172.21.15.58:0/346600577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb000f6700 msgr2=0x7fcb0010bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.488 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.481+0000 7fcb05bc0700 1 --2- 172.21.15.58:0/346600577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb000f6700 0x7fcb0010bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.488 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.481+0000 7fcb05bc0700 1 -- 172.21.15.58:0/346600577 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcafc005040 con 0x7fcb000f5460 2021-11-20T05:34:27.488 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.481+0000 7fcb053bf700 1 --2- 172.21.15.58:0/346600577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb000f6700 0x7fcb0010bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:34:27.489 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.481+0000 7fcb05bc0700 1 --2- 172.21.15.58:0/346600577 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcb000f5460 0x7fcb00109a80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fcafc004e30 tx=0x7fcafc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:27.489 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.481+0000 7fcaf67fc700 1 -- 172.21.15.58:0/346600577 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcafc018070 con 0x7fcb000f5460 2021-11-20T05:34:27.489 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.481+0000 7fcb06bc2700 1 -- 172.21.15.58:0/346600577 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb0010c4d0 con 0x7fcb000f5460 2021-11-20T05:34:27.489 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.481+0000 7fcaf67fc700 1 -- 172.21.15.58:0/346600577 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcafc013410 con 0x7fcb000f5460 2021-11-20T05:34:27.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.481+0000 7fcaf67fc700 1 -- 172.21.15.58:0/346600577 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcafc01c3c0 con 0x7fcb000f5460 2021-11-20T05:34:27.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.481+0000 7fcb06bc2700 1 -- 172.21.15.58:0/346600577 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb001058e0 con 0x7fcb000f5460 2021-11-20T05:34:27.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.482+0000 7fcb06bc2700 1 -- 172.21.15.58:0/346600577 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb000f4850 con 0x7fcb000f5460 2021-11-20T05:34:27.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.483+0000 7fcaf67fc700 1 -- 172.21.15.58:0/346600577 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcafc01d420 con 0x7fcb000f5460 2021-11-20T05:34:27.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.483+0000 7fcaf67fc700 1 --2- 172.21.15.58:0/346600577 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcaec060710 0x7fcaec062bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.491 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.483+0000 7fcaf67fc700 1 -- 172.21.15.58:0/346600577 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fcafc077ad0 con 0x7fcb000f5460 2021-11-20T05:34:27.491 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.483+0000 7fcb053bf700 1 --2- 172.21.15.58:0/346600577 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcaec060710 0x7fcaec062bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.491 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.484+0000 7fcb053bf700 1 --2- 172.21.15.58:0/346600577 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcaec060710 0x7fcaec062bd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fcaf0009740 tx=0x7fcaf0009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:27.492 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.488+0000 7fcaf67fc700 1 -- 172.21.15.58:0/346600577 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcafc04a790 con 0x7fcb000f5460 2021-11-20T05:34:27.689 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.690+0000 7fcb06bc2700 1 -- 172.21.15.58:0/346600577 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb00045ad0 con 0x7fcb000f5460 2021-11-20T05:34:27.690 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.690+0000 7fcaf67fc700 1 -- 172.21.15.58:0/346600577 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+2946 (secure 0 0 0) 0x7fcafc04ddb0 con 0x7fcb000f5460 2021-11-20T05:34:27.691 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:34:27.691 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24259,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2274008158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2274008158},{"type":"v1","addr":"172.21.15.187:6825","nonce":2274008158}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":9},{"gid":24271,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2997529975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2997529975},{"type":"v1","addr":"172.21.15.187:6827","nonce":2997529975}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:34:15.893368+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14428,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14428":{"gid":14428,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.58:6827/3420124164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3420124164},{"type":"v1","addr":"172.21.15.58:6827","nonce":3420124164}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:34:27.692 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.692+0000 7fcb06bc2700 1 -- 172.21.15.58:0/346600577 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcaec060710 msgr2=0x7fcaec062bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.692 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.692+0000 7fcb06bc2700 1 --2- 172.21.15.58:0/346600577 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcaec060710 0x7fcaec062bd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fcaf0009740 tx=0x7fcaf0009be0).stop 2021-11-20T05:34:27.693 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.692+0000 7fcb06bc2700 1 -- 172.21.15.58:0/346600577 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcb000f5460 msgr2=0x7fcb00109a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.693 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.692+0000 7fcb06bc2700 1 --2- 172.21.15.58:0/346600577 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcb000f5460 0x7fcb00109a80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fcafc004e30 tx=0x7fcafc009b30).stop 2021-11-20T05:34:27.693 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.692+0000 7fcb06bc2700 1 -- 172.21.15.58:0/346600577 shutdown_connections 2021-11-20T05:34:27.693 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.692+0000 7fcb06bc2700 1 --2- 172.21.15.58:0/346600577 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcaec060710 0x7fcaec062bd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.694 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.692+0000 7fcb06bc2700 1 --2- 172.21.15.58:0/346600577 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcb000f5460 0x7fcb00109a80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.694 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.692+0000 7fcb06bc2700 1 --2- 172.21.15.58:0/346600577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb000f6700 0x7fcb0010bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.694 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.692+0000 7fcb06bc2700 1 -- 172.21.15.58:0/346600577 >> 172.21.15.58:0/346600577 conn(0x7fcb000f0ba0 msgr2=0x7fcb000fa150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:27.694 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.692+0000 7fcb06bc2700 1 -- 172.21.15.58:0/346600577 shutdown_connections 2021-11-20T05:34:27.694 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.692+0000 7fcb06bc2700 1 -- 172.21.15.58:0/346600577 wait complete. 2021-11-20T05:34:27.695 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 11 2021-11-20T05:34:27.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:27 smithi187 conmon[36358]: audit 2021-11-20T05:34:26.209430+0000 mon.smithi058 (mon.0) 527 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:27.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:27 smithi187 conmon[36358]: cluster 2021-11-20T05:34: 2021-11-20T05:34:27.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:27 smithi187 conmon[36358]: 26.213073+0000 mgr.smithi058.zagdlk (mgr.14164) 202 : cluster [DBG] pgmap v164: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s wr, 6 op/s 2021-11-20T05:34:27.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:27 smithi187 conmon[36358]: audit 2021-11-20T05:34:26.388019+0000 mon.smithi058 (mon.0) 528 : audit [DBG] from='client.? 172.21.15.58:0/140866815' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T05:34:27.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:27 smithi187 conmon[36358]: audit 2021-11-20T05:34:26.697516+0000 mon.smithi058 (mon.0) 529 : audit [DBG] from='client.? 172.21.15.58:0/3578443907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T05:34:27.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:27 smithi187 conmon[36358]: audit 2021-11-20T05:34:27.017551+0000 mon.smithi058 (mon.0) 530 : audit [DBG] from='client.? 172.21.15.58:0/3636422352' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T05:34:27.704 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:34:15.893368+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14428, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14428': {'gid': 14428, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.58:6827/3420124164', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 3420124164}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 3420124164}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:34:27.704 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2021-11-20T05:34:27.705 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.cephfs.smithi058.rxapzd status 2021-11-20T05:34:27.832 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.830+0000 7f6376553700 1 -- 172.21.15.58:0/2414410584 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63700f5fd0 msgr2=0x7f63700f6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.832 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.830+0000 7f6376553700 1 --2- 172.21.15.58:0/2414410584 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63700f5fd0 0x7f63700f6430 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f6364004660 tx=0x7f6364009b30).stop 2021-11-20T05:34:27.832 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.830+0000 7f6376553700 1 -- 172.21.15.58:0/2414410584 shutdown_connections 2021-11-20T05:34:27.832 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.830+0000 7f6376553700 1 --2- 172.21.15.58:0/2414410584 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63700f5fd0 0x7f63700f6430 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.833 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.830+0000 7f6376553700 1 --2- 172.21.15.58:0/2414410584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63700f4d30 0x7f63700f5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.833 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.830+0000 7f6376553700 1 -- 172.21.15.58:0/2414410584 >> 172.21.15.58:0/2414410584 conn(0x7f63700f0470 msgr2=0x7f63700f28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:27.833 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.830+0000 7f6376553700 1 -- 172.21.15.58:0/2414410584 shutdown_connections 2021-11-20T05:34:27.833 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.830+0000 7f6376553700 1 -- 172.21.15.58:0/2414410584 wait complete. 2021-11-20T05:34:27.834 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.830+0000 7f6376553700 1 Processor -- start 2021-11-20T05:34:27.834 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.831+0000 7f6376553700 1 -- start start 2021-11-20T05:34:27.834 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.831+0000 7f6376553700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63700f4d30 0x7f63700fa5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.834 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.831+0000 7f6376553700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63700f5fd0 0x7f63700f8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.835 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.831+0000 7f6376553700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63700fab90 con 0x7f63700f4d30 2021-11-20T05:34:27.835 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.831+0000 7f6376553700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63700f9130 con 0x7f63700f5fd0 2021-11-20T05:34:27.835 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.831+0000 7f636ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63700f4d30 0x7f63700fa5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.835 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.831+0000 7f636ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63700f4d30 0x7f63700fa5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40686/0 (socket says 172.21.15.58:40686) 2021-11-20T05:34:27.835 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.831+0000 7f636ffff700 1 -- 172.21.15.58:0/1933462223 learned_addr learned my addr 172.21.15.58:0/1933462223 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:27.836 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.831+0000 7f636f7fe700 1 --2- 172.21.15.58:0/1933462223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63700f5fd0 0x7f63700f8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.836 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.831+0000 7f636ffff700 1 -- 172.21.15.58:0/1933462223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63700f5fd0 msgr2=0x7f63700f8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.836 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.831+0000 7f636ffff700 1 --2- 172.21.15.58:0/1933462223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63700f5fd0 0x7f63700f8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.836 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.831+0000 7f636ffff700 1 -- 172.21.15.58:0/1933462223 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6364005040 con 0x7f63700f4d30 2021-11-20T05:34:27.837 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.831+0000 7f636ffff700 1 --2- 172.21.15.58:0/1933462223 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63700f4d30 0x7f63700fa5e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f636000e7c0 tx=0x7f636000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:27.837 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.831+0000 7f636cff9700 1 -- 172.21.15.58:0/1933462223 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63600092e0 con 0x7f63700f4d30 2021-11-20T05:34:27.837 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.831+0000 7f636cff9700 1 -- 172.21.15.58:0/1933462223 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f636000f040 con 0x7f63700f4d30 2021-11-20T05:34:27.837 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.831+0000 7f6376553700 1 -- 172.21.15.58:0/1933462223 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63700f93b0 con 0x7f63700f4d30 2021-11-20T05:34:27.837 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.832+0000 7f636cff9700 1 -- 172.21.15.58:0/1933462223 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6360014750 con 0x7f63700f4d30 2021-11-20T05:34:27.838 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.832+0000 7f6376553700 1 -- 172.21.15.58:0/1933462223 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63700f9970 con 0x7f63700f4d30 2021-11-20T05:34:27.845 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.833+0000 7f636cff9700 1 -- 172.21.15.58:0/1933462223 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f636001b030 con 0x7f63700f4d30 2021-11-20T05:34:27.845 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.833+0000 7f636cff9700 1 --2- 172.21.15.58:0/1933462223 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f635805c2a0 0x7f635805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.845 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.833+0000 7f636cff9700 1 -- 172.21.15.58:0/1933462223 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f6360075fa0 con 0x7f63700f4d30 2021-11-20T05:34:27.845 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.833+0000 7f636f7fe700 1 --2- 172.21.15.58:0/1933462223 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f635805c2a0 0x7f635805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.846 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.834+0000 7f636f7fe700 1 --2- 172.21.15.58:0/1933462223 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f635805c2a0 0x7f635805e760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6364004d80 tx=0x7f636400b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:27.846 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.835+0000 7f63880feb80 1 Processor -- start 2021-11-20T05:34:27.846 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.835+0000 7f63880feb80 1 -- start start 2021-11-20T05:34:27.846 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.835+0000 7f63880feb80 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0d9eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.847 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.835+0000 7f63880feb80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 0x5568b0d9fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.847 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.835+0000 7f63880feb80 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b0cdd4d0 con 0x5568b0c0c1d0 2021-11-20T05:34:27.847 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.835+0000 7f63880feb80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b0cf13e0 con 0x5568b0d9efe0 2021-11-20T05:34:27.847 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.835+0000 7f636ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0d9eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.847 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.835+0000 7f636ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0d9eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40692/0 (socket says 172.21.15.58:40692) 2021-11-20T05:34:27.848 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.835+0000 7f636ffff700 1 -- 172.21.15.58:0/3028445528 learned_addr learned my addr 172.21.15.58:0/3028445528 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:27.848 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.835+0000 7f636ffff700 1 -- 172.21.15.58:0/3028445528 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 msgr2=0x5568b0d9fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T05:34:27.848 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.835+0000 7f636ffff700 1 --2- 172.21.15.58:0/3028445528 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 0x5568b0d9fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:34:27.848 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.835+0000 7f636ffff700 1 -- 172.21.15.58:0/3028445528 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568b0d6d3c0 con 0x5568b0c0c1d0 2021-11-20T05:34:27.849 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.836+0000 7f636ffff700 1 --2- 172.21.15.58:0/3028445528 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0d9eaf0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f63600090a0 tx=0x7f6360009540).ready entity=mon.0 client_cookie=7a757995832e6b92 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:27.849 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.836+0000 7f6355ffb700 1 -- 172.21.15.58:0/3028445528 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f636004cb50 con 0x5568b0c0c1d0 2021-11-20T05:34:27.849 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.836+0000 7f6355ffb700 1 -- 172.21.15.58:0/3028445528 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f636004ccb0 con 0x5568b0c0c1d0 2021-11-20T05:34:27.849 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.836+0000 7f6355ffb700 1 -- 172.21.15.58:0/3028445528 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6360007b50 con 0x5568b0c0c1d0 2021-11-20T05:34:27.850 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.836+0000 7f63880feb80 1 -- 172.21.15.58:0/3028445528 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 msgr2=0x5568b0d9eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.850 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.836+0000 7f63880feb80 1 --2- 172.21.15.58:0/3028445528 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0d9eaf0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f63600090a0 tx=0x7f6360009540).stop 2021-11-20T05:34:27.850 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.836+0000 7f63880feb80 1 -- 172.21.15.58:0/3028445528 shutdown_connections 2021-11-20T05:34:27.850 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.836+0000 7f63880feb80 1 --2- 172.21.15.58:0/3028445528 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 0x5568b0d9fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:34:27.850 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.836+0000 7f63880feb80 1 --2- 172.21.15.58:0/3028445528 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0d9eaf0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.851 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.836+0000 7f63880feb80 1 -- 172.21.15.58:0/3028445528 >> 172.21.15.58:0/3028445528 conn(0x5568b0d649b0 msgr2=0x5568b0d99fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:27.852 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.836+0000 7f63880feb80 1 -- 172.21.15.58:0/3028445528 shutdown_connections 2021-11-20T05:34:27.852 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.836+0000 7f63880feb80 1 -- 172.21.15.58:0/3028445528 wait complete. 2021-11-20T05:34:27.852 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.837+0000 7f63880feb80 1 -- start start 2021-11-20T05:34:27.852 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.837+0000 7f63880feb80 1 Processor -- start 2021-11-20T05:34:27.853 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.837+0000 7f63880feb80 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0da78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.853 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.837+0000 7f63880feb80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 0x5568b0da7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.853 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.837+0000 7f63880feb80 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b0cf1520 con 0x5568b0c0c1d0 2021-11-20T05:34:27.853 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.837+0000 7f63880feb80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b0cf1aa0 con 0x5568b0d9efe0 2021-11-20T05:34:27.854 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.837+0000 7f63880feb80 10 client.0 ll_register_callbacks cb 0x5568b0d5e700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-20T05:34:27.854 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.837+0000 7f636ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0da78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.854 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.837+0000 7f636ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0da78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40696/0 (socket says 172.21.15.58:40696) 2021-11-20T05:34:27.854 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.837+0000 7f636ffff700 1 -- 172.21.15.58:0/456536396 learned_addr learned my addr 172.21.15.58:0/456536396 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:27.854 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.837+0000 7f636f7fe700 1 --2- 172.21.15.58:0/456536396 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 0x5568b0da7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.855 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f636ffff700 1 -- 172.21.15.58:0/456536396 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 msgr2=0x5568b0da7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T05:34:27.855 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f636ffff700 1 --2- 172.21.15.58:0/456536396 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 0x5568b0da7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.855 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f636ffff700 1 -- 172.21.15.58:0/456536396 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63700f9970 con 0x5568b0c0c1d0 2021-11-20T05:34:27.855 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f636ffff700 1 --2- 172.21.15.58:0/456536396 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0da78b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f6360053040 tx=0x7f6360009540).ready entity=mon.0 client_cookie=49b2b0283dccb5c1 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:27.856 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f634effd700 10 client.0 ms_handle_connect on v2:172.21.15.58:3300/0 2021-11-20T05:34:27.856 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f634effd700 1 -- 172.21.15.58:0/456536396 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6360007980 con 0x5568b0c0c1d0 2021-11-20T05:34:27.856 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f634effd700 1 -- 172.21.15.58:0/456536396 <== mon.0 v2:172.21.15.58:3300/0 2 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f6360048710 con 0x5568b0c0c1d0 2021-11-20T05:34:27.856 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f63880feb80 1 -- 172.21.15.58:0/456536396 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5568b0d6cce0 con 0x5568b0c0c1d0 2021-11-20T05:34:27.857 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f634effd700 1 -- 172.21.15.58:0/456536396 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_get_version_reply(handle=1 version=11) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6360048920 con 0x5568b0c0c1d0 2021-11-20T05:34:27.857 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f63880feb80 10 client.14508 fetch_fsmap learned FSMap version 11 2021-11-20T05:34:27.857 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f63880feb80 1 -- 172.21.15.58:0/456536396 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,fsmap=11,monmap=3+}) v3 -- 0x5568b0ce0630 con 0x5568b0c0c1d0 2021-11-20T05:34:27.857 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f634effd700 1 -- 172.21.15.58:0/456536396 <== mon.0 v2:172.21.15.58:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x5568b0d6cce0 con 0x5568b0c0c1d0 2021-11-20T05:34:27.857 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f634effd700 1 -- 172.21.15.58:0/456536396 <== mon.0 v2:172.21.15.58:3300/0 5 ==== fsmap(e 11) v1 ==== 1736+0+0 (secure 0 0 0) 0x7f636004a750 con 0x5568b0c0c1d0 2021-11-20T05:34:27.858 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f63880feb80 10 client.14508 fetch_fsmap finished waiting for FSMap version 11 2021-11-20T05:34:27.858 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f63880feb80 10 client.14508 resolve_mds: resolved name 'cephfs.smithi058.rxapzd' to daemon mds.cephfs.smithi058.rxapzd 2021-11-20T05:34:27.858 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f63880feb80 20 client.14508 populate_metadata read hostname 'smithi058' 2021-11-20T05:34:27.858 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f63880feb80 1 --2- 172.21.15.58:0/456536396 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5568b0db0650 0x5568b0db2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f63880feb80 4 client.14508 mds_command: new command op to 14428 tid=0[{"prefix": "get_command_descriptions"}] 2021-11-20T05:34:27.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f63880feb80 1 -- 172.21.15.58:0/456536396 --> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5568b0ce7df0 con 0x5568b0db0650 2021-11-20T05:34:27.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.838+0000 7f636f7fe700 1 --2- 172.21.15.58:0/456536396 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5568b0db0650 0x5568b0db2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.839+0000 7f634effd700 10 client.14508 ms_handle_connect on v2:172.21.15.58:6826/3420124164 2021-11-20T05:34:27.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.839+0000 7f636f7fe700 1 --2- 172.21.15.58:0/456536396 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5568b0db0650 0x5568b0db2a40 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=b572c9a3305fbb13 server_cookie=dbd5e90f021b7d52 in_seq=0 out_seq=0 2021-11-20T05:34:27.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.839+0000 7f634effd700 1 -- 172.21.15.58:0/456536396 <== mds.0 v2:172.21.15.58:6826/3420124164 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x5568b0ce7df0 con 0x5568b0db0650 2021-11-20T05:34:27.861 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.839+0000 7f634effd700 10 client.14508 handle_command_reply: tid=0 2021-11-20T05:34:27.862 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.839+0000 7f63880feb80 1 client.14508 shutdown 2021-11-20T05:34:27.862 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.839+0000 7f63880feb80 1 -- 172.21.15.58:0/456536396 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 msgr2=0x5568b0da78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.862 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.839+0000 7f63880feb80 1 --2- 172.21.15.58:0/456536396 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0da78b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f6360053040 tx=0x7f6360009540).stop 2021-11-20T05:34:27.862 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.839+0000 7f63880feb80 1 -- 172.21.15.58:0/456536396 shutdown_connections 2021-11-20T05:34:27.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.839+0000 7f63880feb80 1 --2- 172.21.15.58:0/456536396 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5568b0db0650 0x5568b0db2a40 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.840+0000 7f63880feb80 1 --2- 172.21.15.58:0/456536396 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0da78b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.840+0000 7f63880feb80 1 --2- 172.21.15.58:0/456536396 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 0x5568b0da7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.840+0000 7f634effd700 0 client.14508 ms_handle_reset on v2:172.21.15.58:6826/3420124164 2021-11-20T05:34:27.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.840+0000 7f63880feb80 1 -- 172.21.15.58:0/456536396 >> 172.21.15.58:0/456536396 conn(0x5568b0d649b0 msgr2=0x5568b0d9b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:27.864 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.840+0000 7f63880feb80 1 -- 172.21.15.58:0/456536396 shutdown_connections 2021-11-20T05:34:27.864 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.840+0000 7f63880feb80 1 -- 172.21.15.58:0/456536396 wait complete. 2021-11-20T05:34:27.864 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.840+0000 7f63880feb80 20 client.14508 trim_cache size 0 max 16384 2021-11-20T05:34:27.871 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.856+0000 7f63880feb80 1 Processor -- start 2021-11-20T05:34:27.871 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.856+0000 7f63880feb80 1 -- start start 2021-11-20T05:34:27.872 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.856+0000 7f63880feb80 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0d971c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.872 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.856+0000 7f63880feb80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 0x5568b0d976b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.872 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.856+0000 7f63880feb80 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b0cbf4a0 con 0x5568b0c0c1d0 2021-11-20T05:34:27.872 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.856+0000 7f63880feb80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b0d6cce0 con 0x5568b0d9efe0 2021-11-20T05:34:27.873 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.856+0000 7f636ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0d971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.873 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.856+0000 7f636ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0d971c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40702/0 (socket says 172.21.15.58:40702) 2021-11-20T05:34:27.873 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.856+0000 7f636ffff700 1 -- 172.21.15.58:0/3802055576 learned_addr learned my addr 172.21.15.58:0/3802055576 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:27.873 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.856+0000 7f636ffff700 1 -- 172.21.15.58:0/3802055576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 msgr2=0x5568b0d976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T05:34:27.873 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.856+0000 7f636f7fe700 1 --2- 172.21.15.58:0/3802055576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 0x5568b0d976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.874 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.856+0000 7f636ffff700 1 --2- 172.21.15.58:0/3802055576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 0x5568b0d976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.874 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.856+0000 7f636ffff700 1 -- 172.21.15.58:0/3802055576 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568b0d99e80 con 0x5568b0c0c1d0 2021-11-20T05:34:27.874 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.856+0000 7f636f7fe700 1 --2- 172.21.15.58:0/3802055576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 0x5568b0d976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:34:27.874 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.857+0000 7f636ffff700 1 --2- 172.21.15.58:0/3802055576 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0d971c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f6360009770 tx=0x7f6360009540).ready entity=mon.0 client_cookie=c6bc6daade994f20 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:27.875 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.857+0000 7f63567fc700 1 -- 172.21.15.58:0/3802055576 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f636004c0d0 con 0x5568b0c0c1d0 2021-11-20T05:34:27.875 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.857+0000 7f63567fc700 1 -- 172.21.15.58:0/3802055576 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6360007980 con 0x5568b0c0c1d0 2021-11-20T05:34:27.875 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.857+0000 7f63567fc700 1 -- 172.21.15.58:0/3802055576 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6360074a20 con 0x5568b0c0c1d0 2021-11-20T05:34:27.875 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.857+0000 7f63880feb80 1 -- 172.21.15.58:0/3802055576 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 msgr2=0x5568b0d971c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.875 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.857+0000 7f63880feb80 1 --2- 172.21.15.58:0/3802055576 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0d971c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f6360009770 tx=0x7f6360009540).stop 2021-11-20T05:34:27.876 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.857+0000 7f63880feb80 1 -- 172.21.15.58:0/3802055576 shutdown_connections 2021-11-20T05:34:27.876 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.857+0000 7f63880feb80 1 --2- 172.21.15.58:0/3802055576 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 0x5568b0d976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.876 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.857+0000 7f63880feb80 1 --2- 172.21.15.58:0/3802055576 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0d971c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.876 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.857+0000 7f63880feb80 1 -- 172.21.15.58:0/3802055576 >> 172.21.15.58:0/3802055576 conn(0x5568b0d649b0 msgr2=0x5568b0da6590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:27.877 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.857+0000 7f63880feb80 1 -- 172.21.15.58:0/3802055576 shutdown_connections 2021-11-20T05:34:27.877 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.857+0000 7f63880feb80 1 -- 172.21.15.58:0/3802055576 wait complete. 2021-11-20T05:34:27.877 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.857+0000 7f63880feb80 1 -- start start 2021-11-20T05:34:27.877 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f63880feb80 1 Processor -- start 2021-11-20T05:34:27.877 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f63880feb80 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0dfc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.878 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f63880feb80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 0x5568b0dfcdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.878 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f63880feb80 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b0dfd370 con 0x5568b0c0c1d0 2021-11-20T05:34:27.878 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f63880feb80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b0dfd4b0 con 0x5568b0d9efe0 2021-11-20T05:34:27.878 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f63880feb80 10 client.0 ll_register_callbacks cb 0x5568b0d5e700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-20T05:34:27.879 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f636ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0dfc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.879 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f636ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0dfc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40706/0 (socket says 172.21.15.58:40706) 2021-11-20T05:34:27.879 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f636ffff700 1 -- 172.21.15.58:0/2975083954 learned_addr learned my addr 172.21.15.58:0/2975083954 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:27.879 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f636f7fe700 1 --2- 172.21.15.58:0/2975083954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 0x5568b0dfcdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.880 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f636ffff700 1 -- 172.21.15.58:0/2975083954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 msgr2=0x5568b0dfcdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T05:34:27.880 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f636ffff700 1 --2- 172.21.15.58:0/2975083954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 0x5568b0dfcdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.880 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f636ffff700 1 -- 172.21.15.58:0/2975083954 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5568b0ce0630 con 0x5568b0c0c1d0 2021-11-20T05:34:27.880 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f636ffff700 1 --2- 172.21.15.58:0/2975083954 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0dfc8d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f6360049390 tx=0x7f6360009540).ready entity=mon.0 client_cookie=a83e009a5c3db447 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:27.880 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f634effd700 10 client.0 ms_handle_connect on v2:172.21.15.58:3300/0 2021-11-20T05:34:27.881 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f634effd700 1 -- 172.21.15.58:0/2975083954 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f636004c080 con 0x5568b0c0c1d0 2021-11-20T05:34:27.881 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f634effd700 1 -- 172.21.15.58:0/2975083954 <== mon.0 v2:172.21.15.58:3300/0 2 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f6360048940 con 0x5568b0c0c1d0 2021-11-20T05:34:27.881 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.858+0000 7f63880feb80 1 -- 172.21.15.58:0/2975083954 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5568b0d99220 con 0x5568b0c0c1d0 2021-11-20T05:34:27.881 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.859+0000 7f634effd700 1 -- 172.21.15.58:0/2975083954 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_get_version_reply(handle=1 version=11) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6360074170 con 0x5568b0c0c1d0 2021-11-20T05:34:27.882 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.859+0000 7f63880feb80 10 client.14512 fetch_fsmap learned FSMap version 11 2021-11-20T05:34:27.882 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.859+0000 7f63880feb80 1 -- 172.21.15.58:0/2975083954 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,fsmap=11,monmap=3+}) v3 -- 0x7f6370111bb0 con 0x5568b0c0c1d0 2021-11-20T05:34:27.882 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.859+0000 7f634effd700 1 -- 172.21.15.58:0/2975083954 <== mon.0 v2:172.21.15.58:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x5568b0d99220 con 0x5568b0c0c1d0 2021-11-20T05:34:27.882 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.859+0000 7f634effd700 1 -- 172.21.15.58:0/2975083954 <== mon.0 v2:172.21.15.58:3300/0 5 ==== fsmap(e 11) v1 ==== 1736+0+0 (secure 0 0 0) 0x7f636004b4d0 con 0x5568b0c0c1d0 2021-11-20T05:34:27.882 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.859+0000 7f63880feb80 10 client.14512 fetch_fsmap finished waiting for FSMap version 11 2021-11-20T05:34:27.883 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.859+0000 7f63880feb80 10 client.14512 resolve_mds: resolved name 'cephfs.smithi058.rxapzd' to daemon mds.cephfs.smithi058.rxapzd 2021-11-20T05:34:27.883 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.859+0000 7f63880feb80 20 client.14512 populate_metadata read hostname 'smithi058' 2021-11-20T05:34:27.883 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.859+0000 7f63880feb80 1 --2- 172.21.15.58:0/2975083954 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5568b0e029e0 0x5568b0e04dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:27.883 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.859+0000 7f63880feb80 4 client.14512 mds_command: new command op to 14428 tid=0[{"prefix": "status"}] 2021-11-20T05:34:27.884 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.859+0000 7f63880feb80 1 -- 172.21.15.58:0/2975083954 --> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5568b0cdf0c0 con 0x5568b0e029e0 2021-11-20T05:34:27.884 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.859+0000 7f636f7fe700 1 --2- 172.21.15.58:0/2975083954 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5568b0e029e0 0x5568b0e04dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:27.884 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.859+0000 7f634effd700 10 client.14512 ms_handle_connect on v2:172.21.15.58:6826/3420124164 2021-11-20T05:34:27.884 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.859+0000 7f636f7fe700 1 --2- 172.21.15.58:0/2975083954 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5568b0e029e0 0x5568b0e04dd0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=4d5e28ae8c0a63e6 server_cookie=9453e56eceaa82c7 in_seq=0 out_seq=0 2021-11-20T05:34:27.884 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.859+0000 7f634effd700 1 -- 172.21.15.58:0/2975083954 <== mds.0 v2:172.21.15.58:6826/3420124164 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5568b0cdf0c0 con 0x5568b0e029e0 2021-11-20T05:34:27.885 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.859+0000 7f634effd700 10 client.14512 handle_command_reply: tid=0 2021-11-20T05:34:27.885 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.860+0000 7f63880feb80 1 client.14512 shutdown 2021-11-20T05:34:27.885 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.860+0000 7f63880feb80 1 -- 172.21.15.58:0/2975083954 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 msgr2=0x5568b0dfc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.885 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.860+0000 7f63880feb80 1 --2- 172.21.15.58:0/2975083954 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0dfc8d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f6360049390 tx=0x7f6360009540).stop 2021-11-20T05:34:27.886 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.860+0000 7f63880feb80 1 -- 172.21.15.58:0/2975083954 shutdown_connections 2021-11-20T05:34:27.886 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.860+0000 7f63880feb80 1 --2- 172.21.15.58:0/2975083954 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5568b0e029e0 0x5568b0e04dd0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.886 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.860+0000 7f63880feb80 1 --2- 172.21.15.58:0/2975083954 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5568b0c0c1d0 0x5568b0dfc8d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.886 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.860+0000 7f63880feb80 1 --2- 172.21.15.58:0/2975083954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5568b0d9efe0 0x5568b0dfcdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.886 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.860+0000 7f634effd700 0 client.14512 ms_handle_reset on v2:172.21.15.58:6826/3420124164 2021-11-20T05:34:27.887 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.860+0000 7f63880feb80 1 -- 172.21.15.58:0/2975083954 >> 172.21.15.58:0/2975083954 conn(0x5568b0d649b0 msgr2=0x5568b0d971c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:27.887 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.860+0000 7f63880feb80 1 -- 172.21.15.58:0/2975083954 shutdown_connections 2021-11-20T05:34:27.887 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.860+0000 7f63880feb80 1 -- 172.21.15.58:0/2975083954 wait complete. 2021-11-20T05:34:27.887 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.860+0000 7f63880feb80 20 client.14512 trim_cache size 0 max 16384 2021-11-20T05:34:27.888 INFO:teuthology.orchestra.run.smithi058.stdout:{ 2021-11-20T05:34:27.888 INFO:teuthology.orchestra.run.smithi058.stdout: "cluster_fsid": "b33f8c4a-49c2-11ec-8c2c-001a4aab830c", 2021-11-20T05:34:27.888 INFO:teuthology.orchestra.run.smithi058.stdout: "whoami": 0, 2021-11-20T05:34:27.889 INFO:teuthology.orchestra.run.smithi058.stdout: "id": 14428, 2021-11-20T05:34:27.889 INFO:teuthology.orchestra.run.smithi058.stdout: "want_state": "up:active", 2021-11-20T05:34:27.889 INFO:teuthology.orchestra.run.smithi058.stdout: "state": "up:active", 2021-11-20T05:34:27.889 INFO:teuthology.orchestra.run.smithi058.stdout: "fs_name": "cephfs", 2021-11-20T05:34:27.890 INFO:teuthology.orchestra.run.smithi058.stdout: "rank_uptime": 13.972879430000001, 2021-11-20T05:34:27.890 INFO:teuthology.orchestra.run.smithi058.stdout: "mdsmap_epoch": 8, 2021-11-20T05:34:27.890 INFO:teuthology.orchestra.run.smithi058.stdout: "osdmap_epoch": 42, 2021-11-20T05:34:27.890 INFO:teuthology.orchestra.run.smithi058.stdout: "osdmap_epoch_barrier": 42, 2021-11-20T05:34:27.890 INFO:teuthology.orchestra.run.smithi058.stdout: "uptime": 18.329534787 2021-11-20T05:34:27.891 INFO:teuthology.orchestra.run.smithi058.stdout:} 2021-11-20T05:34:27.892 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.860+0000 7f6376553700 1 -- 172.21.15.58:0/1933462223 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f635805c2a0 msgr2=0x7f635805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.893 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.860+0000 7f6376553700 1 --2- 172.21.15.58:0/1933462223 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f635805c2a0 0x7f635805e760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6364004d80 tx=0x7f636400b040).stop 2021-11-20T05:34:27.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.860+0000 7f6376553700 1 -- 172.21.15.58:0/1933462223 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63700f4d30 msgr2=0x7f63700fa5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:27.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.860+0000 7f6376553700 1 --2- 172.21.15.58:0/1933462223 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63700f4d30 0x7f63700fa5e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f636000e7c0 tx=0x7f636000b040).stop 2021-11-20T05:34:27.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.861+0000 7f6376553700 1 -- 172.21.15.58:0/1933462223 shutdown_connections 2021-11-20T05:34:27.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.861+0000 7f6376553700 1 --2- 172.21.15.58:0/1933462223 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f635805c2a0 0x7f635805e760 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.861+0000 7f6376553700 1 --2- 172.21.15.58:0/1933462223 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63700f4d30 0x7f63700fa5e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.861+0000 7f6376553700 1 --2- 172.21.15.58:0/1933462223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63700f5fd0 0x7f63700f8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:27.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.861+0000 7f6376553700 1 -- 172.21.15.58:0/1933462223 >> 172.21.15.58:0/1933462223 conn(0x7f63700f0470 msgr2=0x7f637005f180 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:27.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.861+0000 7f6376553700 1 -- 172.21.15.58:0/1933462223 shutdown_connections 2021-11-20T05:34:27.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:27.861+0000 7f6376553700 1 -- 172.21.15.58:0/1933462223 wait complete. 2021-11-20T05:34:27.896 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.cephfs.smithi058.nuxpgf status 2021-11-20T05:34:28.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.020+0000 7f44c907a700 1 -- 172.21.15.58:0/3083957945 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44c40f5580 msgr2=0x7f44c40f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:28.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.020+0000 7f44c907a700 1 --2- 172.21.15.58:0/3083957945 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44c40f5580 0x7f44c40f59a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f44bc0099e0 tx=0x7f44bc004d10).stop 2021-11-20T05:34:28.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.020+0000 7f44c907a700 1 -- 172.21.15.58:0/3083957945 shutdown_connections 2021-11-20T05:34:28.024 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.020+0000 7f44c907a700 1 --2- 172.21.15.58:0/3083957945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44c40f6820 0x7f44c40f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.024 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.020+0000 7f44c907a700 1 --2- 172.21.15.58:0/3083957945 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44c40f5580 0x7f44c40f59a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.024 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.020+0000 7f44c907a700 1 -- 172.21.15.58:0/3083957945 >> 172.21.15.58:0/3083957945 conn(0x7f44c40f0ce0 msgr2=0x7f44c40f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:28.024 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.020+0000 7f44c907a700 1 -- 172.21.15.58:0/3083957945 shutdown_connections 2021-11-20T05:34:28.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.020+0000 7f44c907a700 1 -- 172.21.15.58:0/3083957945 wait complete. 2021-11-20T05:34:28.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.020+0000 7f44c907a700 1 Processor -- start 2021-11-20T05:34:28.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.020+0000 7f44c907a700 1 -- start start 2021-11-20T05:34:28.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44c907a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44c40f5580 0x7f44c4105e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:28.026 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44c907a700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44c40f6820 0x7f44c4106370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:28.026 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44c907a700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c4106920 con 0x7f44c40f6820 2021-11-20T05:34:28.026 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44c907a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c4106a60 con 0x7f44c40f5580 2021-11-20T05:34:28.026 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44c259c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44c40f6820 0x7f44c4106370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:28.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44c259c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44c40f6820 0x7f44c4106370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40718/0 (socket says 172.21.15.58:40718) 2021-11-20T05:34:28.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44c259c700 1 -- 172.21.15.58:0/3008284423 learned_addr learned my addr 172.21.15.58:0/3008284423 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:28.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44c2d9d700 1 --2- 172.21.15.58:0/3008284423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44c40f5580 0x7f44c4105e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:28.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44c259c700 1 -- 172.21.15.58:0/3008284423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44c40f5580 msgr2=0x7f44c4105e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:28.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44c259c700 1 --2- 172.21.15.58:0/3008284423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44c40f5580 0x7f44c4105e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44c259c700 1 -- 172.21.15.58:0/3008284423 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44bc009640 con 0x7f44c40f6820 2021-11-20T05:34:28.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44c2d9d700 1 --2- 172.21.15.58:0/3008284423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44c40f5580 0x7f44c4105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:34:28.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44c259c700 1 --2- 172.21.15.58:0/3008284423 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44c40f6820 0x7f44c4106370 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f44b800e7f0 tx=0x7f44b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:28.029 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44ab7fe700 1 -- 172.21.15.58:0/3008284423 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44b80092e0 con 0x7f44c40f6820 2021-11-20T05:34:28.029 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44ab7fe700 1 -- 172.21.15.58:0/3008284423 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44b800f040 con 0x7f44c40f6820 2021-11-20T05:34:28.029 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44c907a700 1 -- 172.21.15.58:0/3008284423 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c410b500 con 0x7f44c40f6820 2021-11-20T05:34:28.029 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44ab7fe700 1 -- 172.21.15.58:0/3008284423 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44b80146c0 con 0x7f44c40f6820 2021-11-20T05:34:28.030 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.021+0000 7f44c907a700 1 -- 172.21.15.58:0/3008284423 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c410bac0 con 0x7f44c40f6820 2021-11-20T05:34:28.030 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.023+0000 7f44ab7fe700 1 -- 172.21.15.58:0/3008284423 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f44b801b030 con 0x7f44c40f6820 2021-11-20T05:34:28.030 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.023+0000 7f44ab7fe700 1 --2- 172.21.15.58:0/3008284423 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44ac05c2a0 0x7f44ac05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:28.030 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.023+0000 7f44ab7fe700 1 -- 172.21.15.58:0/3008284423 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f44b8075c70 con 0x7f44c40f6820 2021-11-20T05:34:28.031 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.023+0000 7f44c2d9d700 1 --2- 172.21.15.58:0/3008284423 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44ac05c2a0 0x7f44ac05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:28.031 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.023+0000 7f44c2d9d700 1 --2- 172.21.15.58:0/3008284423 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44ac05c2a0 0x7f44ac05e760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f44bc003380 tx=0x7f44bc004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:28.037 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.025+0000 7f44dac25b80 1 Processor -- start 2021-11-20T05:34:28.037 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.025+0000 7f44dac25b80 1 -- start start 2021-11-20T05:34:28.037 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.025+0000 7f44dac25b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31ca91d0 0x560f31e3baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:28.038 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.025+0000 7f44dac25b80 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31e3bfe0 0x560f31e3cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:28.038 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.025+0000 7f44dac25b80 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f31d7a4d0 con 0x560f31e3bfe0 2021-11-20T05:34:28.038 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.025+0000 7f44dac25b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f31d8e3e0 con 0x560f31ca91d0 2021-11-20T05:34:28.038 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.025+0000 7f44c259c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31e3bfe0 0x560f31e3cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:28.039 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.025+0000 7f44c259c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31e3bfe0 0x560f31e3cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40724/0 (socket says 172.21.15.58:40724) 2021-11-20T05:34:28.039 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.025+0000 7f44c259c700 1 -- 172.21.15.58:0/2831367052 learned_addr learned my addr 172.21.15.58:0/2831367052 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:28.039 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.025+0000 7f44c2d9d700 1 --2- 172.21.15.58:0/2831367052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31ca91d0 0x560f31e3baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:28.039 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.025+0000 7f44c259c700 1 -- 172.21.15.58:0/2831367052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31ca91d0 msgr2=0x560f31e3baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T05:34:28.040 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.025+0000 7f44c259c700 1 --2- 172.21.15.58:0/2831367052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31ca91d0 0x560f31e3baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.040 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.025+0000 7f44c259c700 1 -- 172.21.15.58:0/2831367052 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f31e0a3c0 con 0x560f31e3bfe0 2021-11-20T05:34:28.040 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.025+0000 7f44c2d9d700 1 --2- 172.21.15.58:0/2831367052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31ca91d0 0x560f31e3baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:34:28.041 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.025+0000 7f44c259c700 1 --2- 172.21.15.58:0/2831367052 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31e3bfe0 0x560f31e3cd40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f44b8009fa0 tx=0x7f44b8048a30).ready entity=mon.0 client_cookie=dc7e0c043c993976 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:28.041 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.026+0000 7f44a8be5700 1 -- 172.21.15.58:0/2831367052 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44b8048f30 con 0x560f31e3bfe0 2021-11-20T05:34:28.041 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.026+0000 7f44a8be5700 1 -- 172.21.15.58:0/2831367052 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44b80077b0 con 0x560f31e3bfe0 2021-11-20T05:34:28.041 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.026+0000 7f44a8be5700 1 -- 172.21.15.58:0/2831367052 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44b80746f0 con 0x560f31e3bfe0 2021-11-20T05:34:28.041 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.026+0000 7f44dac25b80 1 -- 172.21.15.58:0/2831367052 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31e3bfe0 msgr2=0x560f31e3cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:28.042 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.026+0000 7f44dac25b80 1 --2- 172.21.15.58:0/2831367052 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31e3bfe0 0x560f31e3cd40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f44b8009fa0 tx=0x7f44b8048a30).stop 2021-11-20T05:34:28.042 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.026+0000 7f44dac25b80 1 -- 172.21.15.58:0/2831367052 shutdown_connections 2021-11-20T05:34:28.042 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.026+0000 7f44dac25b80 1 --2- 172.21.15.58:0/2831367052 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31e3bfe0 0x560f31e3cd40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.042 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.026+0000 7f44dac25b80 1 --2- 172.21.15.58:0/2831367052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31ca91d0 0x560f31e3baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.043 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.026+0000 7f44dac25b80 1 -- 172.21.15.58:0/2831367052 >> 172.21.15.58:0/2831367052 conn(0x560f31e019b0 msgr2=0x560f31e36fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:28.043 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.026+0000 7f44dac25b80 1 -- 172.21.15.58:0/2831367052 shutdown_connections 2021-11-20T05:34:28.043 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.026+0000 7f44dac25b80 1 -- 172.21.15.58:0/2831367052 wait complete. 2021-11-20T05:34:28.043 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.027+0000 7f44dac25b80 1 -- start start 2021-11-20T05:34:28.044 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.027+0000 7f44dac25b80 1 Processor -- start 2021-11-20T05:34:28.044 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.027+0000 7f44dac25b80 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e46a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:28.044 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.027+0000 7f44dac25b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31e3bfe0 0x560f31e46f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:28.044 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.027+0000 7f44dac25b80 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f31d8e520 con 0x560f31ca91d0 2021-11-20T05:34:28.044 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.027+0000 7f44dac25b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f31d8eaa0 con 0x560f31e3bfe0 2021-11-20T05:34:28.045 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.027+0000 7f44dac25b80 10 client.0 ll_register_callbacks cb 0x560f31dfb700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-20T05:34:28.045 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.027+0000 7f44c2d9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e46a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:28.045 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.027+0000 7f44c2d9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e46a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40726/0 (socket says 172.21.15.58:40726) 2021-11-20T05:34:28.045 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.027+0000 7f44c2d9d700 1 -- 172.21.15.58:0/1098647338 learned_addr learned my addr 172.21.15.58:0/1098647338 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:28.046 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.027+0000 7f44c259c700 1 --2- 172.21.15.58:0/1098647338 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31e3bfe0 0x560f31e46f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:28.046 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.027+0000 7f44c2d9d700 1 -- 172.21.15.58:0/1098647338 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31e3bfe0 msgr2=0x560f31e46f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T05:34:28.046 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.027+0000 7f44c2d9d700 1 --2- 172.21.15.58:0/1098647338 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31e3bfe0 0x560f31e46f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.046 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.027+0000 7f44c2d9d700 1 -- 172.21.15.58:0/1098647338 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c410bac0 con 0x560f31ca91d0 2021-11-20T05:34:28.047 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.028+0000 7f44c2d9d700 1 --2- 172.21.15.58:0/1098647338 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e46a90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f44c40f6680 tx=0x7f44bc01c000).ready entity=mon.0 client_cookie=32cc468a3725faf1 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:28.047 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.028+0000 7f44a1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.58:3300/0 2021-11-20T05:34:28.047 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.028+0000 7f44a1ffb700 1 -- 172.21.15.58:0/1098647338 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44bc024070 con 0x560f31ca91d0 2021-11-20T05:34:28.047 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.028+0000 7f44a1ffb700 1 -- 172.21.15.58:0/1098647338 <== mon.0 v2:172.21.15.58:3300/0 2 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f44bc007e10 con 0x560f31ca91d0 2021-11-20T05:34:28.048 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.028+0000 7f44dac25b80 1 -- 172.21.15.58:0/1098647338 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560f31e09ce0 con 0x560f31ca91d0 2021-11-20T05:34:28.048 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.028+0000 7f44a1ffb700 1 -- 172.21.15.58:0/1098647338 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_get_version_reply(handle=1 version=11) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44bc013630 con 0x560f31ca91d0 2021-11-20T05:34:28.048 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.028+0000 7f44dac25b80 10 client.14520 fetch_fsmap learned FSMap version 11 2021-11-20T05:34:28.048 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.028+0000 7f44dac25b80 1 -- 172.21.15.58:0/1098647338 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,fsmap=11,monmap=3+}) v3 -- 0x560f31d7d630 con 0x560f31ca91d0 2021-11-20T05:34:28.049 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.028+0000 7f44a1ffb700 1 -- 172.21.15.58:0/1098647338 <== mon.0 v2:172.21.15.58:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x560f31e09ce0 con 0x560f31ca91d0 2021-11-20T05:34:28.049 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.028+0000 7f44a1ffb700 1 -- 172.21.15.58:0/1098647338 <== mon.0 v2:172.21.15.58:3300/0 5 ==== fsmap(e 11) v1 ==== 1736+0+0 (secure 0 0 0) 0x7f44bc033020 con 0x560f31ca91d0 2021-11-20T05:34:28.049 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.028+0000 7f44dac25b80 10 client.14520 fetch_fsmap finished waiting for FSMap version 11 2021-11-20T05:34:28.049 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.028+0000 7f44dac25b80 10 client.14520 resolve_mds: resolved name 'cephfs.smithi058.nuxpgf' to daemon mds.cephfs.smithi058.nuxpgf 2021-11-20T05:34:28.050 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.028+0000 7f44dac25b80 20 client.14520 populate_metadata read hostname 'smithi058' 2021-11-20T05:34:28.050 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.028+0000 7f44dac25b80 1 --2- 172.21.15.58:0/1098647338 >> [v2:172.21.15.58:6828/4151830882,v1:172.21.15.58:6829/4151830882] conn(0x560f31e4f820 0x560f31e51c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:28.050 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.028+0000 7f44dac25b80 4 client.14520 mds_command: new command op to 14444 tid=0[{"prefix": "get_command_descriptions"}] 2021-11-20T05:34:28.050 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.028+0000 7f44dac25b80 1 -- 172.21.15.58:0/1098647338 --> [v2:172.21.15.58:6828/4151830882,v1:172.21.15.58:6829/4151830882] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560f31d84df0 con 0x560f31e4f820 2021-11-20T05:34:28.050 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.028+0000 7f44c259c700 1 --2- 172.21.15.58:0/1098647338 >> [v2:172.21.15.58:6828/4151830882,v1:172.21.15.58:6829/4151830882] conn(0x560f31e4f820 0x560f31e51c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:28.051 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.029+0000 7f44a1ffb700 10 client.14520 ms_handle_connect on v2:172.21.15.58:6828/4151830882 2021-11-20T05:34:28.051 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.029+0000 7f44c259c700 1 --2- 172.21.15.58:0/1098647338 >> [v2:172.21.15.58:6828/4151830882,v1:172.21.15.58:6829/4151830882] conn(0x560f31e4f820 0x560f31e51c10 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.1 client_cookie=9b39dfd17c4551d3 server_cookie=410c70852b2ddec4 in_seq=0 out_seq=0 2021-11-20T05:34:28.051 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.029+0000 7f44a1ffb700 1 -- 172.21.15.58:0/1098647338 <== mds.1 v2:172.21.15.58:6828/4151830882 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x560f31d84df0 con 0x560f31e4f820 2021-11-20T05:34:28.051 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.029+0000 7f44a1ffb700 10 client.14520 handle_command_reply: tid=0 2021-11-20T05:34:28.052 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.029+0000 7f44dac25b80 1 client.14520 shutdown 2021-11-20T05:34:28.052 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.029+0000 7f44dac25b80 1 -- 172.21.15.58:0/1098647338 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 msgr2=0x560f31e46a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:28.052 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.029+0000 7f44dac25b80 1 --2- 172.21.15.58:0/1098647338 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e46a90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f44c40f6680 tx=0x7f44bc01c000).stop 2021-11-20T05:34:28.052 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.029+0000 7f44dac25b80 1 -- 172.21.15.58:0/1098647338 shutdown_connections 2021-11-20T05:34:28.052 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.029+0000 7f44dac25b80 1 --2- 172.21.15.58:0/1098647338 >> [v2:172.21.15.58:6828/4151830882,v1:172.21.15.58:6829/4151830882] conn(0x560f31e4f820 0x560f31e51c10 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.053 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.029+0000 7f44dac25b80 1 --2- 172.21.15.58:0/1098647338 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e46a90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.053 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.029+0000 7f44dac25b80 1 --2- 172.21.15.58:0/1098647338 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31e3bfe0 0x560f31e46f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.053 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.029+0000 7f44a1ffb700 0 client.14520 ms_handle_reset on v2:172.21.15.58:6828/4151830882 2021-11-20T05:34:28.053 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.029+0000 7f44dac25b80 1 -- 172.21.15.58:0/1098647338 >> 172.21.15.58:0/1098647338 conn(0x560f31e019b0 msgr2=0x560f31e381d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:28.054 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.030+0000 7f44dac25b80 1 -- 172.21.15.58:0/1098647338 shutdown_connections 2021-11-20T05:34:28.054 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.030+0000 7f44dac25b80 1 -- 172.21.15.58:0/1098647338 wait complete. 2021-11-20T05:34:28.054 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.030+0000 7f44dac25b80 20 client.14520 trim_cache size 0 max 16384 2021-11-20T05:34:28.061 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.049+0000 7f44dac25b80 1 Processor -- start 2021-11-20T05:34:28.061 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.050+0000 7f44dac25b80 1 -- start start 2021-11-20T05:34:28.061 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.050+0000 7f44dac25b80 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e348d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:28.062 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.050+0000 7f44dac25b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31e3bfe0 0x560f31e34dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:28.062 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.050+0000 7f44dac25b80 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f31d5c4a0 con 0x560f31ca91d0 2021-11-20T05:34:28.062 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.050+0000 7f44dac25b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f31d84df0 con 0x560f31e3bfe0 2021-11-20T05:34:28.062 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.050+0000 7f44c2d9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e348d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:28.063 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.050+0000 7f44c2d9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e348d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40732/0 (socket says 172.21.15.58:40732) 2021-11-20T05:34:28.063 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.050+0000 7f44c2d9d700 1 -- 172.21.15.58:0/2737126048 learned_addr learned my addr 172.21.15.58:0/2737126048 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:28.063 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.050+0000 7f44c259c700 1 --2- 172.21.15.58:0/2737126048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31e3bfe0 0x560f31e34dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:28.063 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.050+0000 7f44c2d9d700 1 -- 172.21.15.58:0/2737126048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31e3bfe0 msgr2=0x560f31e34dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T05:34:28.063 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.050+0000 7f44c2d9d700 1 --2- 172.21.15.58:0/2737126048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31e3bfe0 0x560f31e34dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.064 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.050+0000 7f44c2d9d700 1 -- 172.21.15.58:0/2737126048 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f31e46510 con 0x560f31ca91d0 2021-11-20T05:34:28.064 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.050+0000 7f44c2d9d700 1 --2- 172.21.15.58:0/2737126048 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e348d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f44bc004a60 tx=0x7f44bc01c000).ready entity=mon.0 client_cookie=6f6e8ab8c6f27f06 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:28.064 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.050+0000 7f44a93e6700 1 -- 172.21.15.58:0/2737126048 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44bc024040 con 0x560f31ca91d0 2021-11-20T05:34:28.064 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.051+0000 7f44a93e6700 1 -- 172.21.15.58:0/2737126048 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44bc019040 con 0x560f31ca91d0 2021-11-20T05:34:28.065 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.051+0000 7f44a93e6700 1 -- 172.21.15.58:0/2737126048 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44bc013630 con 0x560f31ca91d0 2021-11-20T05:34:28.065 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.051+0000 7f44dac25b80 1 -- 172.21.15.58:0/2737126048 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 msgr2=0x560f31e348d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:28.065 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.051+0000 7f44dac25b80 1 --2- 172.21.15.58:0/2737126048 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e348d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f44bc004a60 tx=0x7f44bc01c000).stop 2021-11-20T05:34:28.065 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.051+0000 7f44dac25b80 1 -- 172.21.15.58:0/2737126048 shutdown_connections 2021-11-20T05:34:28.066 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.051+0000 7f44dac25b80 1 --2- 172.21.15.58:0/2737126048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31e3bfe0 0x560f31e34dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.066 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.051+0000 7f44dac25b80 1 --2- 172.21.15.58:0/2737126048 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e348d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.066 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.051+0000 7f44dac25b80 1 -- 172.21.15.58:0/2737126048 >> 172.21.15.58:0/2737126048 conn(0x560f31e019b0 msgr2=0x560f31e341c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:28.066 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.051+0000 7f44dac25b80 1 -- 172.21.15.58:0/2737126048 shutdown_connections 2021-11-20T05:34:28.066 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.051+0000 7f44dac25b80 1 -- 172.21.15.58:0/2737126048 wait complete. 2021-11-20T05:34:28.067 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.052+0000 7f44dac25b80 1 -- start start 2021-11-20T05:34:28.067 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.052+0000 7f44dac25b80 1 Processor -- start 2021-11-20T05:34:28.067 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.052+0000 7f44dac25b80 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e9b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:28.067 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.052+0000 7f44dac25b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31e3bfe0 0x560f31e9bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:28.068 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.052+0000 7f44dac25b80 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f31e9c470 con 0x560f31ca91d0 2021-11-20T05:34:28.068 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.052+0000 7f44dac25b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f31e9c5b0 con 0x560f31e3bfe0 2021-11-20T05:34:28.068 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.052+0000 7f44dac25b80 10 client.0 ll_register_callbacks cb 0x560f31dfb700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-20T05:34:28.068 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.052+0000 7f44c2d9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e9b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:28.068 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.052+0000 7f44c2d9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e9b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40736/0 (socket says 172.21.15.58:40736) 2021-11-20T05:34:28.069 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.052+0000 7f44c2d9d700 1 -- 172.21.15.58:0/4029210484 learned_addr learned my addr 172.21.15.58:0/4029210484 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:28.069 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.052+0000 7f44c259c700 1 --2- 172.21.15.58:0/4029210484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31e3bfe0 0x560f31e9bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:28.069 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.052+0000 7f44c2d9d700 1 -- 172.21.15.58:0/4029210484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31e3bfe0 msgr2=0x560f31e9bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T05:34:28.069 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.052+0000 7f44c2d9d700 1 --2- 172.21.15.58:0/4029210484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31e3bfe0 0x560f31e9bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.070 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.052+0000 7f44c2d9d700 1 -- 172.21.15.58:0/4029210484 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f31d7d630 con 0x560f31ca91d0 2021-11-20T05:34:28.070 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.052+0000 7f44c2d9d700 1 --2- 172.21.15.58:0/4029210484 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e9b9d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f44bc01cb20 tx=0x7f44bc01c000).ready entity=mon.0 client_cookie=cb4b3373bf145ae7 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:28.070 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.052+0000 7f44a1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.58:3300/0 2021-11-20T05:34:28.070 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.053+0000 7f44a1ffb700 1 -- 172.21.15.58:0/4029210484 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44bc013840 con 0x560f31ca91d0 2021-11-20T05:34:28.071 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.053+0000 7f44a1ffb700 1 -- 172.21.15.58:0/4029210484 <== mon.0 v2:172.21.15.58:3300/0 2 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f44bc0139a0 con 0x560f31ca91d0 2021-11-20T05:34:28.071 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.053+0000 7f44dac25b80 1 -- 172.21.15.58:0/4029210484 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560f31e36220 con 0x560f31ca91d0 2021-11-20T05:34:28.071 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.053+0000 7f44a1ffb700 1 -- 172.21.15.58:0/4029210484 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_get_version_reply(handle=1 version=11) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44bc013b60 con 0x560f31ca91d0 2021-11-20T05:34:28.071 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.053+0000 7f44dac25b80 10 client.14524 fetch_fsmap learned FSMap version 11 2021-11-20T05:34:28.071 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.053+0000 7f44dac25b80 1 -- 172.21.15.58:0/4029210484 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,fsmap=11,monmap=3+}) v3 -- 0x7f44c410d8e0 con 0x560f31ca91d0 2021-11-20T05:34:28.072 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.053+0000 7f44a1ffb700 1 -- 172.21.15.58:0/4029210484 <== mon.0 v2:172.21.15.58:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x560f31e36220 con 0x560f31ca91d0 2021-11-20T05:34:28.072 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.053+0000 7f44a1ffb700 1 -- 172.21.15.58:0/4029210484 <== mon.0 v2:172.21.15.58:3300/0 5 ==== fsmap(e 11) v1 ==== 1736+0+0 (secure 0 0 0) 0x7f44bc033070 con 0x560f31ca91d0 2021-11-20T05:34:28.072 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.053+0000 7f44dac25b80 10 client.14524 fetch_fsmap finished waiting for FSMap version 11 2021-11-20T05:34:28.072 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.053+0000 7f44dac25b80 10 client.14524 resolve_mds: resolved name 'cephfs.smithi058.nuxpgf' to daemon mds.cephfs.smithi058.nuxpgf 2021-11-20T05:34:28.073 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.053+0000 7f44dac25b80 20 client.14524 populate_metadata read hostname 'smithi058' 2021-11-20T05:34:28.073 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.053+0000 7f44dac25b80 1 --2- 172.21.15.58:0/4029210484 >> [v2:172.21.15.58:6828/4151830882,v1:172.21.15.58:6829/4151830882] conn(0x560f31ea1c70 0x560f31ea4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:28.073 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.053+0000 7f44dac25b80 4 client.14524 mds_command: new command op to 14444 tid=0[{"prefix": "status"}] 2021-11-20T05:34:28.073 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.053+0000 7f44dac25b80 1 -- 172.21.15.58:0/4029210484 --> [v2:172.21.15.58:6828/4151830882,v1:172.21.15.58:6829/4151830882] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560f31d7c0c0 con 0x560f31ea1c70 2021-11-20T05:34:28.074 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.053+0000 7f44c259c700 1 --2- 172.21.15.58:0/4029210484 >> [v2:172.21.15.58:6828/4151830882,v1:172.21.15.58:6829/4151830882] conn(0x560f31ea1c70 0x560f31ea4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:28.074 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.054+0000 7f44a1ffb700 10 client.14524 ms_handle_connect on v2:172.21.15.58:6828/4151830882 2021-11-20T05:34:28.074 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.054+0000 7f44c259c700 1 --2- 172.21.15.58:0/4029210484 >> [v2:172.21.15.58:6828/4151830882,v1:172.21.15.58:6829/4151830882] conn(0x560f31ea1c70 0x560f31ea4060 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.1 client_cookie=2cd65f6e28cd9116 server_cookie=40cdc27205fbd406 in_seq=0 out_seq=0 2021-11-20T05:34:28.074 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.054+0000 7f44a1ffb700 1 -- 172.21.15.58:0/4029210484 <== mds.1 v2:172.21.15.58:6828/4151830882 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x560f31d7c0c0 con 0x560f31ea1c70 2021-11-20T05:34:28.074 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.054+0000 7f44a1ffb700 10 client.14524 handle_command_reply: tid=0 2021-11-20T05:34:28.075 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.054+0000 7f44dac25b80 1 client.14524 shutdown 2021-11-20T05:34:28.075 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.054+0000 7f44dac25b80 1 -- 172.21.15.58:0/4029210484 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 msgr2=0x560f31e9b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:28.075 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.054+0000 7f44dac25b80 1 --2- 172.21.15.58:0/4029210484 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e9b9d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f44bc01cb20 tx=0x7f44bc01c000).stop 2021-11-20T05:34:28.075 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.054+0000 7f44dac25b80 1 -- 172.21.15.58:0/4029210484 shutdown_connections 2021-11-20T05:34:28.076 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.054+0000 7f44dac25b80 1 --2- 172.21.15.58:0/4029210484 >> [v2:172.21.15.58:6828/4151830882,v1:172.21.15.58:6829/4151830882] conn(0x560f31ea1c70 0x560f31ea4060 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.076 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.054+0000 7f44dac25b80 1 --2- 172.21.15.58:0/4029210484 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x560f31ca91d0 0x560f31e9b9d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.076 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.054+0000 7f44a1ffb700 0 client.14524 ms_handle_reset on v2:172.21.15.58:6828/4151830882 2021-11-20T05:34:28.076 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.054+0000 7f44dac25b80 1 --2- 172.21.15.58:0/4029210484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560f31e3bfe0 0x560f31e9bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.076 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.054+0000 7f44dac25b80 1 -- 172.21.15.58:0/4029210484 >> 172.21.15.58:0/4029210484 conn(0x560f31e019b0 msgr2=0x560f31e349e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:28.077 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.054+0000 7f44dac25b80 1 -- 172.21.15.58:0/4029210484 shutdown_connections 2021-11-20T05:34:28.077 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.054+0000 7f44dac25b80 1 -- 172.21.15.58:0/4029210484 wait complete. 2021-11-20T05:34:28.077 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.054+0000 7f44dac25b80 20 client.14524 trim_cache size 0 max 16384 2021-11-20T05:34:28.078 INFO:teuthology.orchestra.run.smithi058.stdout:{ 2021-11-20T05:34:28.078 INFO:teuthology.orchestra.run.smithi058.stdout: "cluster_fsid": "b33f8c4a-49c2-11ec-8c2c-001a4aab830c", 2021-11-20T05:34:28.078 INFO:teuthology.orchestra.run.smithi058.stdout: "whoami": 1, 2021-11-20T05:34:28.078 INFO:teuthology.orchestra.run.smithi058.stdout: "id": 14444, 2021-11-20T05:34:28.079 INFO:teuthology.orchestra.run.smithi058.stdout: "want_state": "up:active", 2021-11-20T05:34:28.079 INFO:teuthology.orchestra.run.smithi058.stdout: "state": "up:active", 2021-11-20T05:34:28.079 INFO:teuthology.orchestra.run.smithi058.stdout: "fs_name": "cephfs", 2021-11-20T05:34:28.079 INFO:teuthology.orchestra.run.smithi058.stdout: "rank_uptime": 13.159737777, 2021-11-20T05:34:28.080 INFO:teuthology.orchestra.run.smithi058.stdout: "mdsmap_epoch": 8, 2021-11-20T05:34:28.080 INFO:teuthology.orchestra.run.smithi058.stdout: "osdmap_epoch": 42, 2021-11-20T05:34:28.080 INFO:teuthology.orchestra.run.smithi058.stdout: "osdmap_epoch_barrier": 42, 2021-11-20T05:34:28.080 INFO:teuthology.orchestra.run.smithi058.stdout: "uptime": 13.829483301 2021-11-20T05:34:28.081 INFO:teuthology.orchestra.run.smithi058.stdout:} 2021-11-20T05:34:28.082 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.055+0000 7f44c907a700 1 -- 172.21.15.58:0/3008284423 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44ac05c2a0 msgr2=0x7f44ac05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:28.083 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.055+0000 7f44c907a700 1 --2- 172.21.15.58:0/3008284423 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44ac05c2a0 0x7f44ac05e760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f44bc003380 tx=0x7f44bc004d10).stop 2021-11-20T05:34:28.083 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.055+0000 7f44c907a700 1 -- 172.21.15.58:0/3008284423 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44c40f6820 msgr2=0x7f44c4106370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:28.083 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.055+0000 7f44c907a700 1 --2- 172.21.15.58:0/3008284423 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44c40f6820 0x7f44c4106370 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f44b800e7f0 tx=0x7f44b800b040).stop 2021-11-20T05:34:28.083 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.055+0000 7f44c907a700 1 -- 172.21.15.58:0/3008284423 shutdown_connections 2021-11-20T05:34:28.084 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.055+0000 7f44c907a700 1 --2- 172.21.15.58:0/3008284423 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44ac05c2a0 0x7f44ac05e760 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.084 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.055+0000 7f44c907a700 1 --2- 172.21.15.58:0/3008284423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44c40f5580 0x7f44c4105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.084 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.055+0000 7f44c907a700 1 --2- 172.21.15.58:0/3008284423 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44c40f6820 0x7f44c4106370 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:28.084 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.055+0000 7f44c907a700 1 -- 172.21.15.58:0/3008284423 >> 172.21.15.58:0/3008284423 conn(0x7f44c40f0ce0 msgr2=0x7f44c40fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:28.084 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.055+0000 7f44c907a700 1 -- 172.21.15.58:0/3008284423 shutdown_connections 2021-11-20T05:34:28.085 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:28.055+0000 7f44c907a700 1 -- 172.21.15.58:0/3008284423 wait complete. 2021-11-20T05:34:28.085 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2021-11-20T05:34:28.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:28.086 DEBUG:teuthology.orchestra.run.smithi058:> mkdir -p /home/ubuntu/cephtest/mnt.0 2021-11-20T05:34:28.105 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-11-20T05:34:28.106 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2021-11-20T05:34:28.195 INFO:teuthology.orchestra.run.smithi058.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2021-11-20T05:34:28.196 INFO:teuthology.orchestra.run.smithi058.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2021-11-20T05:34:28.319 INFO:teuthology.orchestra.run.smithi058.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2021-11-20T05:34:28.320 INFO:teuthology.orchestra.run.smithi058.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2021-11-20T05:34:28.324 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:28 smithi058 conmon[33537]: audit 2021-11-20T05:34:27.330270+0000 mon.smithi058 (mon.0) 2021-11-20T05:34:28.324 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:28 smithi058 conmon[33537]: 531 : audit [DBG] from='client.? 172.21.15.58:0/348289470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T05:34:28.324 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:28 smithi058 conmon[33537]: audit 2021-11-20T05:34:27.691382+0000 mon.smithi058 (mon.0) 532 : audit [DBG] from='client.? 172.21.15.58:0/346600577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:34:28.326 INFO:tasks.cephfs.kernel_mount:mount command passed 2021-11-20T05:34:28.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:28.327 DEBUG:teuthology.orchestra.run.smithi058:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2021-11-20T05:34:28.364 INFO:tasks.kclient:client.1 config is {} 2021-11-20T05:34:28.365 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2021-11-20T05:34:28.365 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2021-11-20T05:34:28.366 INFO:tasks.cephfs.mount:self.client.name = client.1 2021-11-20T05:34:28.366 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2021-11-20T05:34:28.366 INFO:tasks.cephfs.mount:self.cephfs_name = None 2021-11-20T05:34:28.367 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2021-11-20T05:34:28.367 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2021-11-20T05:34:28.367 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2021-11-20T05:34:28.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:28.368 DEBUG:teuthology.orchestra.run.smithi187:> ip addr 2021-11-20T05:34:28.388 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2021-11-20T05:34:28.388 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2021-11-20T05:34:28.389 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2021-11-20T05:34:28.389 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2021-11-20T05:34:28.389 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2021-11-20T05:34:28.389 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2021-11-20T05:34:28.390 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T05:34:28.390 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2021-11-20T05:34:28.390 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T05:34:28.390 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2021-11-20T05:34:28.391 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T05:34:28.391 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2021-11-20T05:34:28.391 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2021-11-20T05:34:28.391 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2021-11-20T05:34:28.391 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2021-11-20T05:34:28.392 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42223sec preferred_lft 42223sec 2021-11-20T05:34:28.392 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link noprefixroute 2021-11-20T05:34:28.392 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2021-11-20T05:34:28.392 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2021-11-20T05:34:28.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:28.393 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec bash -c ' 2021-11-20T05:34:28.394 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2021-11-20T05:34:28.394 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link add name ceph-brx type bridge 2021-11-20T05:34:28.394 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip addr flush dev ceph-brx 2021-11-20T05:34:28.395 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link set ceph-brx up 2021-11-20T05:34:28.395 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2021-11-20T05:34:28.395 DEBUG:teuthology.orchestra.run.smithi187:> ') 2021-11-20T05:34:28.472 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:28 smithi187 conmon[36358]: audit 2021-11-20T05:34:27.330270+0000 mon.smithi058 (mon.0) 531 : audit [DBG] from='client.? 172.21.15.58:0/348289470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T05:34:28.473 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:28 smithi187 conmon[36358]: audit 2021-11-20T05:34:27.691382+0000 mon.smithi058 (mon.0) 532 : audit [DBG] from='client.? 172.21.15.58:0/346600577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:34:28.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:28.564 DEBUG:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2021-11-20T05:34:28.590 INFO:teuthology.orchestra.run.smithi187.stdout:1 2021-11-20T05:34:28.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:28.592 DEBUG:teuthology.orchestra.run.smithi187:> route 2021-11-20T05:34:28.667 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2021-11-20T05:34:28.667 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2021-11-20T05:34:28.667 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2021-11-20T05:34:28.668 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2021-11-20T05:34:28.668 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2021-11-20T05:34:28.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:28.669 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec bash -c ' 2021-11-20T05:34:28.669 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2021-11-20T05:34:28.669 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2021-11-20T05:34:28.670 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2021-11-20T05:34:28.670 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2021-11-20T05:34:28.670 DEBUG:teuthology.orchestra.run.smithi187:> ') 2021-11-20T05:34:28.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:28.937 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list 2021-11-20T05:34:28.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:28.955 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list-id 2021-11-20T05:34:29.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:29.010 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec bash -c ' 2021-11-20T05:34:29.010 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2021-11-20T05:34:29.011 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2021-11-20T05:34:29.011 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2021-11-20T05:34:29.011 DEBUG:teuthology.orchestra.run.smithi187:> ') 2021-11-20T05:34:29.101 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2021-11-20T05:34:29.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:29.101 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec bash -c ' 2021-11-20T05:34:29.102 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2021-11-20T05:34:29.102 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2021-11-20T05:34:29.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2021-11-20T05:34:29.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2021-11-20T05:34:29.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2021-11-20T05:34:29.104 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2021-11-20T05:34:29.104 DEBUG:teuthology.orchestra.run.smithi187:> ') 2021-11-20T05:34:29.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T05:34:29.336 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec bash -c ' 2021-11-20T05:34:29.336 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2021-11-20T05:34:29.337 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2021-11-20T05:34:29.337 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link set dev brx.0 master ceph-brx 2021-11-20T05:34:29.337 DEBUG:teuthology.orchestra.run.smithi187:> ') 2021-11-20T05:34:29.372 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:29 smithi187 conmon[36358]: cluster 2021-11-20T05:34:28.214162+0000 mgr.smithi058.zagdlk (mgr.14164) 203 : cluster [DBG] pgmap v165: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s wr, 4 op/s 2021-11-20T05:34:29.373 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:29 smithi187 conmon[36358]: audit 2021-11-20T05:34:28.525248+0000 mon.smithi058 (mon.0) 533 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:29.373 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:29 smithi187 conmon[36358]: audit 2021-11-20T05:34:28.528556+0000 mon.smithi058 (mon.0) 534 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:29.440 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-11-20T05:34:29.539 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:29 smithi058 conmon[33537]: cluster 2021-11-20T05:34:28.214162+0000 mgr.smithi058.zagdlk (mgr.14164) 203 : cluster [DBG] pgmap v165: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s wr, 4 op/s 2021-11-20T05:34:29.540 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:29 smithi058 conmon[33537]: audit 2021-11-20T05:34:28.525248+0000 mon.smithi058 (mon.0) 533 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:29.540 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:29 smithi058 conmon[33537]: audit 2021-11-20T05:34 2021-11-20T05:34:29.540 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:29 smithi058 conmon[33537]: :28.528556+0000 mon.smithi058 (mon.0) 534 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:34:29.587 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.585+0000 7fc9bf482700 1 -- 172.21.15.58:0/1277388363 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc9b8060140 msgr2=0x7fc9b80fb680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:29.588 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.585+0000 7fc9bf482700 1 --2- 172.21.15.58:0/1277388363 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc9b8060140 0x7fc9b80fb680 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fc9b0009a30 tx=0x7fc9b0004c50).stop 2021-11-20T05:34:29.588 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.585+0000 7fc9bf482700 1 -- 172.21.15.58:0/1277388363 shutdown_connections 2021-11-20T05:34:29.588 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.585+0000 7fc9bf482700 1 --2- 172.21.15.58:0/1277388363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9b80fbb70 0x7fc9b80fdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:29.588 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.585+0000 7fc9bf482700 1 --2- 172.21.15.58:0/1277388363 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc9b8060140 0x7fc9b80fb680 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:29.588 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.585+0000 7fc9bf482700 1 -- 172.21.15.58:0/1277388363 >> 172.21.15.58:0/1277388363 conn(0x7fc9b80f0ce0 msgr2=0x7fc9b80f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:29.589 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.586+0000 7fc9bf482700 1 -- 172.21.15.58:0/1277388363 shutdown_connections 2021-11-20T05:34:29.589 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.586+0000 7fc9bf482700 1 -- 172.21.15.58:0/1277388363 wait complete. 2021-11-20T05:34:29.589 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.586+0000 7fc9bf482700 1 Processor -- start 2021-11-20T05:34:29.590 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.586+0000 7fc9bf482700 1 -- start start 2021-11-20T05:34:29.590 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.586+0000 7fc9bf482700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9b8060140 0x7fc9b8109040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:29.590 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.586+0000 7fc9bf482700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc9b80fbb70 0x7fc9b8109530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:29.590 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.586+0000 7fc9bf482700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b8109ae0 con 0x7fc9b80fbb70 2021-11-20T05:34:29.591 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.586+0000 7fc9bf482700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b8109c20 con 0x7fc9b8060140 2021-11-20T05:34:29.591 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.586+0000 7fc9bca1d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc9b80fbb70 0x7fc9b8109530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:29.591 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.586+0000 7fc9bca1d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc9b80fbb70 0x7fc9b8109530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40754/0 (socket says 172.21.15.58:40754) 2021-11-20T05:34:29.591 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.586+0000 7fc9bca1d700 1 -- 172.21.15.58:0/603748962 learned_addr learned my addr 172.21.15.58:0/603748962 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:29.592 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.586+0000 7fc9bd21e700 1 --2- 172.21.15.58:0/603748962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9b8060140 0x7fc9b8109040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:29.592 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.587+0000 7fc9bca1d700 1 -- 172.21.15.58:0/603748962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9b8060140 msgr2=0x7fc9b8109040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:29.592 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.587+0000 7fc9bca1d700 1 --2- 172.21.15.58:0/603748962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9b8060140 0x7fc9b8109040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:29.592 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.587+0000 7fc9bca1d700 1 -- 172.21.15.58:0/603748962 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ac005040 con 0x7fc9b80fbb70 2021-11-20T05:34:29.592 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.587+0000 7fc9bca1d700 1 --2- 172.21.15.58:0/603748962 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc9b80fbb70 0x7fc9b8109530 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fc9ac004930 tx=0x7fc9ac009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:29.593 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.587+0000 7fc9b5ffb700 1 -- 172.21.15.58:0/603748962 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9ac00e030 con 0x7fc9b80fbb70 2021-11-20T05:34:29.593 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.587+0000 7fc9bf482700 1 -- 172.21.15.58:0/603748962 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b0009640 con 0x7fc9b80fbb70 2021-11-20T05:34:29.593 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.587+0000 7fc9b5ffb700 1 -- 172.21.15.58:0/603748962 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc9ac00f040 con 0x7fc9b80fbb70 2021-11-20T05:34:29.593 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.587+0000 7fc9b5ffb700 1 -- 172.21.15.58:0/603748962 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9ac014830 con 0x7fc9b80fbb70 2021-11-20T05:34:29.594 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.587+0000 7fc9bf482700 1 -- 172.21.15.58:0/603748962 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b810eaf0 con 0x7fc9b80fbb70 2021-11-20T05:34:29.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.588+0000 7fc9bf482700 1 -- 172.21.15.58:0/603748962 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b8046330 con 0x7fc9b80fbb70 2021-11-20T05:34:29.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.588+0000 7fc9b5ffb700 1 -- 172.21.15.58:0/603748962 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc9ac010430 con 0x7fc9b80fbb70 2021-11-20T05:34:29.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.589+0000 7fc9b5ffb700 1 --2- 172.21.15.58:0/603748962 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc9a005c2a0 0x7fc9a005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:29.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.589+0000 7fc9b5ffb700 1 -- 172.21.15.58:0/603748962 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fc9ac0765c0 con 0x7fc9b80fbb70 2021-11-20T05:34:29.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.589+0000 7fc9bd21e700 1 --2- 172.21.15.58:0/603748962 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc9a005c2a0 0x7fc9a005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:29.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.589+0000 7fc9bd21e700 1 --2- 172.21.15.58:0/603748962 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc9a005c2a0 0x7fc9a005e760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc9b0003380 tx=0x7fc9b0004c50).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:29.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.593+0000 7fc9b5ffb700 1 -- 172.21.15.58:0/603748962 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc9ac049280 con 0x7fc9b80fbb70 2021-11-20T05:34:29.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.749+0000 7fc9bf482700 1 -- 172.21.15.58:0/603748962 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9b8027a00 con 0x7fc9b80fbb70 2021-11-20T05:34:29.750 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.750+0000 7fc9b5ffb700 1 -- 172.21.15.58:0/603748962 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+14084 (secure 0 0 0) 0x7fc9ac04c8a0 con 0x7fc9b80fbb70 2021-11-20T05:34:29.750 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:34:29.750 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":42,"fsid":"b33f8c4a-49c2-11ec-8c2c-001a4aab830c","created":"2021-11-20T05:28:54.787342+0000","modified":"2021-11-20T05:34:08.318974+0000","last_up_change":"2021-11-20T05:33:10.122001+0000","last_in_change":"2021-11-20T05:32:53.577246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T05:32:27.068478+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2021-11-20T05:34:04.784086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2021-11-20T05:34:05.738224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e2f5103a-e0d2-4bc3-88e1-f28420c98071","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6802","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6803","nonce":2375873844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6804","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6805","nonce":2375873844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6808","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6809","nonce":2375873844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6806","nonce":2375873844},{"type":"v1","addr":"172.21.15.58:6807","nonce":2375873844}]},"public_addr":"172.21.15.58:6803/2375873844","cluster_addr":"172.21.15.58:6805/2375873844","heartbeat_back_addr":"172.21.15.58:6809/2375873844","heartbeat_front_addr":"172.21.15.58:6807/2375873844","state":["exists","up"]},{"osd":1,"uuid":"e0243e40-c76e-4c68-8ac0-8f555035e948","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6810","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6811","nonce":3875723869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6812","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6813","nonce":3875723869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6816","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6817","nonce":3875723869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6814","nonce":3875723869},{"type":"v1","addr":"172.21.15.58:6815","nonce":3875723869}]},"public_addr":"172.21.15.58:6811/3875723869","cluster_addr":"172.21.15.58:6813/3875723869","heartbeat_back_addr":"172.21.15.58:6817/3875723869","heartbeat_front_addr":"172.21.15.58:6815/3875723869","state":["exists","up"]},{"osd":2,"uuid":"9b35ec54-f60f-49c5-ab1b-d2c5b5eb72e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6818","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6819","nonce":745015076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6820","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6821","nonce":745015076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6824","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6825","nonce":745015076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6822","nonce":745015076},{"type":"v1","addr":"172.21.15.58:6823","nonce":745015076}]},"public_addr":"172.21.15.58:6819/745015076","cluster_addr":"172.21.15.58:6821/745015076","heartbeat_back_addr":"172.21.15.58:6825/745015076","heartbeat_front_addr":"172.21.15.58:6823/745015076","state":["exists","up"]},{"osd":3,"uuid":"8390f146-b7e1-4039-83d3-7ed1c832dc8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6801","nonce":1941886381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6803","nonce":1941886381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6807","nonce":1941886381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1941886381},{"type":"v1","addr":"172.21.15.187:6805","nonce":1941886381}]},"public_addr":"172.21.15.187:6801/1941886381","cluster_addr":"172.21.15.187:6803/1941886381","heartbeat_back_addr":"172.21.15.187:6807/1941886381","heartbeat_front_addr":"172.21.15.187:6805/1941886381","state":["exists","up"]},{"osd":4,"uuid":"89dc0c2f-c52c-48ca-b3ab-792e299d494d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6809","nonce":3483552299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6811","nonce":3483552299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6815","nonce":3483552299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3483552299},{"type":"v1","addr":"172.21.15.187:6813","nonce":3483552299}]},"public_addr":"172.21.15.187:6809/3483552299","cluster_addr":"172.21.15.187:6811/3483552299","heartbeat_back_addr":"172.21.15.187:6815/3483552299","heartbeat_front_addr":"172.21.15.187:6813/3483552299","state":["exists","up"]},{"osd":5,"uuid":"8190ded3-7b78-4d71-9c23-e5bc4112fecd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6817","nonce":566077514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6819","nonce":566077514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6823","nonce":566077514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":566077514},{"type":"v1","addr":"172.21.15.187:6821","nonce":566077514}]},"public_addr":"172.21.15.187:6817/566077514","cluster_addr":"172.21.15.187:6819/566077514","heartbeat_back_addr":"172.21.15.187:6823/566077514","heartbeat_front_addr":"172.21.15.187:6821/566077514","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:31:38.522095+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:31:56.685873+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:15.227272+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:31.493769+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:32:50.070298+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T05:33:08.140441+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.58:6801/2422532259":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/1616356583":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/2030112153":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/3301163446":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6801/1261811851":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6800/1261811851":"2021-11-21T05:29:42.633679+0000","172.21.15.58:0/1447224424":"2021-11-21T05:29:42.633679+0000","172.21.15.58:6800/324085383":"2021-11-21T05:29:22.266837+0000","172.21.15.58:6801/324085383":"2021-11-21T05:29:22.266837+0000","172.21.15.58:0/574129054":"2021-11-21T05:29:22.266837+0000","172.21.15.58:6800/2422532259":"2021-11-21T05:29:59.311707+0000","172.21.15.58:0/2571399354":"2021-11-21T05:29:22.266837+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T05:34:29.752 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.752+0000 7fc9bf482700 1 -- 172.21.15.58:0/603748962 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc9a005c2a0 msgr2=0x7fc9a005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:29.752 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.752+0000 7fc9bf482700 1 --2- 172.21.15.58:0/603748962 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc9a005c2a0 0x7fc9a005e760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc9b0003380 tx=0x7fc9b0004c50).stop 2021-11-20T05:34:29.752 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.752+0000 7fc9bf482700 1 -- 172.21.15.58:0/603748962 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc9b80fbb70 msgr2=0x7fc9b8109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:29.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.752+0000 7fc9bf482700 1 --2- 172.21.15.58:0/603748962 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc9b80fbb70 0x7fc9b8109530 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fc9ac004930 tx=0x7fc9ac009b30).stop 2021-11-20T05:34:29.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.752+0000 7fc9bf482700 1 -- 172.21.15.58:0/603748962 shutdown_connections 2021-11-20T05:34:29.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.752+0000 7fc9bf482700 1 --2- 172.21.15.58:0/603748962 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc9a005c2a0 0x7fc9a005e760 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:29.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.752+0000 7fc9bf482700 1 --2- 172.21.15.58:0/603748962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9b8060140 0x7fc9b8109040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:29.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.752+0000 7fc9bf482700 1 --2- 172.21.15.58:0/603748962 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc9b80fbb70 0x7fc9b8109530 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:29.754 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.752+0000 7fc9bf482700 1 -- 172.21.15.58:0/603748962 >> 172.21.15.58:0/603748962 conn(0x7fc9b80f0ce0 msgr2=0x7fc9b80f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:29.754 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.752+0000 7fc9bf482700 1 -- 172.21.15.58:0/603748962 shutdown_connections 2021-11-20T05:34:29.754 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.752+0000 7fc9bf482700 1 -- 172.21.15.58:0/603748962 wait complete. 2021-11-20T05:34:29.764 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-20T05:32:27.068478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2021-11-20T05:34:04.784086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2021-11-20T05:34:05.738224+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-11-20T05:34:29.765 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2021-11-20T05:34:29.888 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.886+0000 7f3f29101700 1 -- 172.21.15.58:0/3682853915 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3f240f7920 msgr2=0x7f3f240f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:29.889 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.886+0000 7f3f29101700 1 --2- 172.21.15.58:0/3682853915 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3f240f7920 0x7f3f240f7d80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f3f0c004660 tx=0x7f3f0c009b30).stop 2021-11-20T05:34:29.889 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.886+0000 7f3f29101700 1 -- 172.21.15.58:0/3682853915 shutdown_connections 2021-11-20T05:34:29.889 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.886+0000 7f3f29101700 1 --2- 172.21.15.58:0/3682853915 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3f240f7920 0x7f3f240f7d80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:29.890 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.886+0000 7f3f29101700 1 --2- 172.21.15.58:0/3682853915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f240f6680 0x7f3f240f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:29.890 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.886+0000 7f3f29101700 1 -- 172.21.15.58:0/3682853915 >> 172.21.15.58:0/3682853915 conn(0x7f3f240f1de0 msgr2=0x7f3f240f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:29.890 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.887+0000 7f3f29101700 1 -- 172.21.15.58:0/3682853915 shutdown_connections 2021-11-20T05:34:29.890 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.887+0000 7f3f29101700 1 -- 172.21.15.58:0/3682853915 wait complete. 2021-11-20T05:34:29.890 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.887+0000 7f3f29101700 1 Processor -- start 2021-11-20T05:34:29.891 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.887+0000 7f3f29101700 1 -- start start 2021-11-20T05:34:29.891 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.887+0000 7f3f29101700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3f240f6680 0x7f3f24106e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:29.891 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.887+0000 7f3f29101700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f240f7920 0x7f3f24107360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:29.891 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.887+0000 7f3f29101700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f24107910 con 0x7f3f240f6680 2021-11-20T05:34:29.892 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.887+0000 7f3f29101700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f24107a50 con 0x7f3f240f7920 2021-11-20T05:34:29.892 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.887+0000 7f3f22d9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3f240f6680 0x7f3f24106e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:29.892 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.887+0000 7f3f22d9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3f240f6680 0x7f3f24106e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40762/0 (socket says 172.21.15.58:40762) 2021-11-20T05:34:29.892 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.887+0000 7f3f22d9d700 1 -- 172.21.15.58:0/3197259930 learned_addr learned my addr 172.21.15.58:0/3197259930 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:29.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.888+0000 7f3f22d9d700 1 -- 172.21.15.58:0/3197259930 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f240f7920 msgr2=0x7f3f24107360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:29.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.888+0000 7f3f22d9d700 1 --2- 172.21.15.58:0/3197259930 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f240f7920 0x7f3f24107360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:34:29.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.888+0000 7f3f22d9d700 1 -- 172.21.15.58:0/3197259930 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c005040 con 0x7f3f240f6680 2021-11-20T05:34:29.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.888+0000 7f3f22d9d700 1 --2- 172.21.15.58:0/3197259930 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3f240f6680 0x7f3f24106e70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f3f1c002700 tx=0x7f3f1c009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:29.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.888+0000 7f3f1b7fe700 1 -- 172.21.15.58:0/3197259930 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f1c009320 con 0x7f3f240f6680 2021-11-20T05:34:29.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.888+0000 7f3f1b7fe700 1 -- 172.21.15.58:0/3197259930 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f1c00f040 con 0x7f3f240f6680 2021-11-20T05:34:29.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.888+0000 7f3f29101700 1 -- 172.21.15.58:0/3197259930 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2410c4f0 con 0x7f3f240f6680 2021-11-20T05:34:29.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.888+0000 7f3f1b7fe700 1 -- 172.21.15.58:0/3197259930 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f1c014900 con 0x7f3f240f6680 2021-11-20T05:34:29.896 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.888+0000 7f3f29101700 1 -- 172.21.15.58:0/3197259930 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2410cab0 con 0x7f3f240f6680 2021-11-20T05:34:29.896 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.889+0000 7f3f29101700 1 -- 172.21.15.58:0/3197259930 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f24027a00 con 0x7f3f240f6680 2021-11-20T05:34:29.896 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.889+0000 7f3f1b7fe700 1 -- 172.21.15.58:0/3197259930 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3f1c009480 con 0x7f3f240f6680 2021-11-20T05:34:29.897 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.890+0000 7f3f1b7fe700 1 --2- 172.21.15.58:0/3197259930 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3f1005c2a0 0x7f3f1005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:29.898 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.890+0000 7f3f1b7fe700 1 -- 172.21.15.58:0/3197259930 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f3f1c076220 con 0x7f3f240f6680 2021-11-20T05:34:29.898 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.890+0000 7f3f2259c700 1 --2- 172.21.15.58:0/3197259930 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3f1005c2a0 0x7f3f1005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:29.898 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.890+0000 7f3f2259c700 1 --2- 172.21.15.58:0/3197259930 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3f1005c2a0 0x7f3f1005e760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f3f0c009f50 tx=0x7f3f0c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:29.898 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:29.894+0000 7f3f1b7fe700 1 -- 172.21.15.58:0/3197259930 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f1c048ee0 con 0x7f3f240f6680 2021-11-20T05:34:30.055 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.054+0000 7f3f29101700 1 -- 172.21.15.58:0/3197259930 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f3f240fee10 con 0x7f3f240f6680 2021-11-20T05:34:30.055 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.055+0000 7f3f1b7fe700 1 -- 172.21.15.58:0/3197259930 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v42) v1 ==== 110+0+10 (secure 0 0 0) 0x7f3f1c04c500 con 0x7f3f240f6680 2021-11-20T05:34:30.056 INFO:teuthology.orchestra.run.smithi058.stdout:pg_num: 1 2021-11-20T05:34:30.059 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.059+0000 7f3f29101700 1 -- 172.21.15.58:0/3197259930 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3f1005c2a0 msgr2=0x7f3f1005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:30.060 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.059+0000 7f3f29101700 1 --2- 172.21.15.58:0/3197259930 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3f1005c2a0 0x7f3f1005e760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f3f0c009f50 tx=0x7f3f0c009b30).stop 2021-11-20T05:34:30.060 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.059+0000 7f3f29101700 1 -- 172.21.15.58:0/3197259930 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3f240f6680 msgr2=0x7f3f24106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:30.060 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.059+0000 7f3f29101700 1 --2- 172.21.15.58:0/3197259930 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3f240f6680 0x7f3f24106e70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f3f1c002700 tx=0x7f3f1c009d80).stop 2021-11-20T05:34:30.061 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.060+0000 7f3f29101700 1 -- 172.21.15.58:0/3197259930 shutdown_connections 2021-11-20T05:34:30.061 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.060+0000 7f3f29101700 1 --2- 172.21.15.58:0/3197259930 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3f1005c2a0 0x7f3f1005e760 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:30.061 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.060+0000 7f3f29101700 1 --2- 172.21.15.58:0/3197259930 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3f240f6680 0x7f3f24106e70 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:30.061 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.060+0000 7f3f29101700 1 --2- 172.21.15.58:0/3197259930 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f240f7920 0x7f3f24107360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:34:30.061 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.060+0000 7f3f29101700 1 -- 172.21.15.58:0/3197259930 >> 172.21.15.58:0/3197259930 conn(0x7f3f240f1de0 msgr2=0x7f3f240fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:30.062 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.060+0000 7f3f29101700 1 -- 172.21.15.58:0/3197259930 shutdown_connections 2021-11-20T05:34:30.062 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.060+0000 7f3f29101700 1 -- 172.21.15.58:0/3197259930 wait complete. 2021-11-20T05:34:30.075 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2021-11-20T05:34:30.200 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.197+0000 7f186db42700 1 -- 172.21.15.58:0/580524306 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18680f5580 msgr2=0x7f18680f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:30.200 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.197+0000 7f186db42700 1 --2- 172.21.15.58:0/580524306 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18680f5580 0x7f18680f59a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f1858004660 tx=0x7f1858009b30).stop 2021-11-20T05:34:30.200 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.197+0000 7f186db42700 1 -- 172.21.15.58:0/580524306 shutdown_connections 2021-11-20T05:34:30.200 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.197+0000 7f186db42700 1 --2- 172.21.15.58:0/580524306 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18680f6820 0x7f18680f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:30.201 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.197+0000 7f186db42700 1 --2- 172.21.15.58:0/580524306 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18680f5580 0x7f18680f59a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:30.201 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.197+0000 7f186db42700 1 -- 172.21.15.58:0/580524306 >> 172.21.15.58:0/580524306 conn(0x7f18680f0ce0 msgr2=0x7f18680f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:30.201 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.198+0000 7f186db42700 1 -- 172.21.15.58:0/580524306 shutdown_connections 2021-11-20T05:34:30.201 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.198+0000 7f186db42700 1 -- 172.21.15.58:0/580524306 wait complete. 2021-11-20T05:34:30.202 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.198+0000 7f186db42700 1 Processor -- start 2021-11-20T05:34:30.202 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.198+0000 7f186db42700 1 -- start start 2021-11-20T05:34:30.202 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.198+0000 7f186db42700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18680f5580 0x7f18680604c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:30.202 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.198+0000 7f186db42700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18680f6820 0x7f186810bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:30.203 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.198+0000 7f186db42700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186810c170 con 0x7f18680f6820 2021-11-20T05:34:30.203 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.198+0000 7f186db42700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186810c2b0 con 0x7f18680f5580 2021-11-20T05:34:30.203 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.199+0000 7f1866ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18680f6820 0x7f186810bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:30.203 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.199+0000 7f1866ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18680f6820 0x7f186810bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40780/0 (socket says 172.21.15.58:40780) 2021-11-20T05:34:30.204 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.199+0000 7f1866ffd700 1 -- 172.21.15.58:0/1071919579 learned_addr learned my addr 172.21.15.58:0/1071919579 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:30.204 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.199+0000 7f18677fe700 1 --2- 172.21.15.58:0/1071919579 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18680f5580 0x7f18680604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:30.204 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.199+0000 7f1866ffd700 1 -- 172.21.15.58:0/1071919579 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18680f5580 msgr2=0x7f18680604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:30.204 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.199+0000 7f1866ffd700 1 --2- 172.21.15.58:0/1071919579 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18680f5580 0x7f18680604c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:30.204 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.199+0000 7f1866ffd700 1 -- 172.21.15.58:0/1071919579 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1858005040 con 0x7f18680f6820 2021-11-20T05:34:30.205 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.199+0000 7f1866ffd700 1 --2- 172.21.15.58:0/1071919579 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18680f6820 0x7f186810bc80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f185c002700 tx=0x7f185c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:30.205 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.199+0000 7f184ffff700 1 -- 172.21.15.58:0/1071919579 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f185c009320 con 0x7f18680f6820 2021-11-20T05:34:30.205 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.199+0000 7f184ffff700 1 -- 172.21.15.58:0/1071919579 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f185c00f040 con 0x7f18680f6820 2021-11-20T05:34:30.206 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.199+0000 7f186db42700 1 -- 172.21.15.58:0/1071919579 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186810c510 con 0x7f18680f6820 2021-11-20T05:34:30.206 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.199+0000 7f184ffff700 1 -- 172.21.15.58:0/1071919579 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f185c0147e0 con 0x7f18680f6820 2021-11-20T05:34:30.206 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.199+0000 7f186db42700 1 -- 172.21.15.58:0/1071919579 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f186810cad0 con 0x7f18680f6820 2021-11-20T05:34:30.206 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.200+0000 7f186db42700 1 -- 172.21.15.58:0/1071919579 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1868027a00 con 0x7f18680f6820 2021-11-20T05:34:30.207 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.200+0000 7f184ffff700 1 -- 172.21.15.58:0/1071919579 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f185c009480 con 0x7f18680f6820 2021-11-20T05:34:30.208 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.201+0000 7f184ffff700 1 --2- 172.21.15.58:0/1071919579 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f185005c2a0 0x7f185005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:30.208 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.201+0000 7f184ffff700 1 -- 172.21.15.58:0/1071919579 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f185c076420 con 0x7f18680f6820 2021-11-20T05:34:30.208 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.201+0000 7f18677fe700 1 --2- 172.21.15.58:0/1071919579 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f185005c2a0 0x7f185005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:30.209 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.201+0000 7f18677fe700 1 --2- 172.21.15.58:0/1071919579 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f185005c2a0 0x7f185005e760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f1858005010 tx=0x7f1858009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:30.209 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.205+0000 7f184ffff700 1 -- 172.21.15.58:0/1071919579 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f185c049160 con 0x7f18680f6820 2021-11-20T05:34:30.369 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.370+0000 7f186db42700 1 -- 172.21.15.58:0/1071919579 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f18680fdd10 con 0x7f18680f6820 2021-11-20T05:34:30.370 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.370+0000 7f184ffff700 1 -- 172.21.15.58:0/1071919579 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v42) v1 ==== 107+0+11 (secure 0 0 0) 0x7f185c04c780 con 0x7f18680f6820 2021-11-20T05:34:30.370 INFO:teuthology.orchestra.run.smithi058.stdout:pg_num: 32 2021-11-20T05:34:30.372 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.372+0000 7f186db42700 1 -- 172.21.15.58:0/1071919579 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f185005c2a0 msgr2=0x7f185005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:30.373 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.372+0000 7f186db42700 1 --2- 172.21.15.58:0/1071919579 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f185005c2a0 0x7f185005e760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f1858005010 tx=0x7f1858009b30).stop 2021-11-20T05:34:30.374 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.372+0000 7f186db42700 1 -- 172.21.15.58:0/1071919579 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18680f6820 msgr2=0x7f186810bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:30.376 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.372+0000 7f186db42700 1 --2- 172.21.15.58:0/1071919579 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18680f6820 0x7f186810bc80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f185c002700 tx=0x7f185c009de0).stop 2021-11-20T05:34:30.376 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.372+0000 7f186db42700 1 -- 172.21.15.58:0/1071919579 shutdown_connections 2021-11-20T05:34:30.377 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.372+0000 7f186db42700 1 --2- 172.21.15.58:0/1071919579 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f185005c2a0 0x7f185005e760 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:30.377 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.372+0000 7f186db42700 1 --2- 172.21.15.58:0/1071919579 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18680f5580 0x7f18680604c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:30.377 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.372+0000 7f186db42700 1 --2- 172.21.15.58:0/1071919579 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18680f6820 0x7f186810bc80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:30.377 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.372+0000 7f186db42700 1 -- 172.21.15.58:0/1071919579 >> 172.21.15.58:0/1071919579 conn(0x7f18680f0ce0 msgr2=0x7f18680fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:30.378 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.372+0000 7f186db42700 1 -- 172.21.15.58:0/1071919579 shutdown_connections 2021-11-20T05:34:30.378 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.372+0000 7f186db42700 1 -- 172.21.15.58:0/1071919579 wait complete. 2021-11-20T05:34:30.383 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:30 smithi058 conmon[33537]: audit 2021-11-20T05:34:29.751005+0000 mon.smithi058 (mon 2021-11-20T05:34:30.383 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:30 smithi058 conmon[33537]: .0) 535 : audit [DBG] from='client.? 172.21.15.58:0/603748962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T05:34:30.383 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:30 smithi058 conmon[33537]: audit 2021-11-20T05:34:30.056065+0000 mon.smithi058 (mon.0) 536 : audit [DBG] from='client.? 172.21.15.58:0/3197259930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T05:34:30.386 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2021-11-20T05:34:30.511 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.508+0000 7f838ed18700 1 -- 172.21.15.58:0/1921155340 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83880f5880 msgr2=0x7f83880f5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:30.511 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.508+0000 7f838ed18700 1 --2- 172.21.15.58:0/1921155340 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83880f5880 0x7f83880f5d00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f8374004660 tx=0x7f8374009b30).stop 2021-11-20T05:34:30.511 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.508+0000 7f838ed18700 1 -- 172.21.15.58:0/1921155340 shutdown_connections 2021-11-20T05:34:30.512 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.508+0000 7f838ed18700 1 --2- 172.21.15.58:0/1921155340 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83880f5880 0x7f83880f5d00 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:30.512 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.508+0000 7f838ed18700 1 --2- 172.21.15.58:0/1921155340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83880f6b80 0x7f83880f6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:30.512 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.508+0000 7f838ed18700 1 -- 172.21.15.58:0/1921155340 >> 172.21.15.58:0/1921155340 conn(0x7f83880f0470 msgr2=0x7f83880f28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:30.512 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.508+0000 7f838ed18700 1 -- 172.21.15.58:0/1921155340 shutdown_connections 2021-11-20T05:34:30.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.508+0000 7f838ed18700 1 -- 172.21.15.58:0/1921155340 wait complete. 2021-11-20T05:34:30.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.508+0000 7f838ed18700 1 Processor -- start 2021-11-20T05:34:30.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.508+0000 7f838ed18700 1 -- start start 2021-11-20T05:34:30.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f838ed18700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83880f5880 0x7f83880f5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:30.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f838ed18700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83880f6b80 0x7f838810bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:30.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f838ed18700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838810c1d0 con 0x7f83880f6b80 2021-11-20T05:34:30.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f838ed18700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838810c310 con 0x7f83880f5880 2021-11-20T05:34:30.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f8387fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83880f6b80 0x7f838810bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:30.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f8387fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83880f6b80 0x7f838810bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40792/0 (socket says 172.21.15.58:40792) 2021-11-20T05:34:30.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f8387fff700 1 -- 172.21.15.58:0/812116793 learned_addr learned my addr 172.21.15.58:0/812116793 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:30.516 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f838cab4700 1 --2- 172.21.15.58:0/812116793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83880f5880 0x7f83880f5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:30.517 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f8387fff700 1 -- 172.21.15.58:0/812116793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83880f5880 msgr2=0x7f83880f5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:30.517 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f8387fff700 1 --2- 172.21.15.58:0/812116793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83880f5880 0x7f83880f5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:30.517 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f8387fff700 1 -- 172.21.15.58:0/812116793 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8374005040 con 0x7f83880f6b80 2021-11-20T05:34:30.517 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f838cab4700 1 --2- 172.21.15.58:0/812116793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83880f5880 0x7f83880f5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:34:30.518 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f8387fff700 1 --2- 172.21.15.58:0/812116793 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83880f6b80 0x7f838810bce0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f8374007a70 tx=0x7f8374009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:30.518 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f83857fa700 1 -- 172.21.15.58:0/812116793 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8374018070 con 0x7f83880f6b80 2021-11-20T05:34:30.518 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f83857fa700 1 -- 172.21.15.58:0/812116793 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8374009700 con 0x7f83880f6b80 2021-11-20T05:34:30.518 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f83857fa700 1 -- 172.21.15.58:0/812116793 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f837401c620 con 0x7f83880f6b80 2021-11-20T05:34:30.519 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f838ed18700 1 -- 172.21.15.58:0/812116793 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f838810c510 con 0x7f83880f6b80 2021-11-20T05:34:30.519 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.509+0000 7f838ed18700 1 -- 172.21.15.58:0/812116793 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838810ca40 con 0x7f83880f6b80 2021-11-20T05:34:30.519 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.510+0000 7f838ed18700 1 -- 172.21.15.58:0/812116793 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8388046330 con 0x7f83880f6b80 2021-11-20T05:34:30.519 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.511+0000 7f83857fa700 1 -- 172.21.15.58:0/812116793 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f837401d420 con 0x7f83880f6b80 2021-11-20T05:34:30.520 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.511+0000 7f83857fa700 1 --2- 172.21.15.58:0/812116793 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f837805c2a0 0x7f837805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:30.520 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.511+0000 7f83857fa700 1 -- 172.21.15.58:0/812116793 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f8374077d00 con 0x7f83880f6b80 2021-11-20T05:34:30.520 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.511+0000 7f838cab4700 1 --2- 172.21.15.58:0/812116793 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f837805c2a0 0x7f837805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:30.521 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.512+0000 7f838cab4700 1 --2- 172.21.15.58:0/812116793 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f837805c2a0 0x7f837805e760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f838805f4e0 tx=0x7f837c009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:30.521 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.515+0000 7f83857fa700 1 -- 172.21.15.58:0/812116793 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f837407b030 con 0x7f83880f6b80 2021-11-20T05:34:30.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.672+0000 7f838ed18700 1 -- 172.21.15.58:0/812116793 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f8388027a00 con 0x7f83880f6b80 2021-11-20T05:34:30.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.672+0000 7f83857fa700 1 -- 172.21.15.58:0/812116793 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v42) v1 ==== 107+0+11 (secure 0 0 0) 0x7f837404aa40 con 0x7f83880f6b80 2021-11-20T05:34:30.673 INFO:teuthology.orchestra.run.smithi058.stdout:pg_num: 32 2021-11-20T05:34:30.675 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.674+0000 7f838ed18700 1 -- 172.21.15.58:0/812116793 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f837805c2a0 msgr2=0x7f837805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:30.675 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.674+0000 7f838ed18700 1 --2- 172.21.15.58:0/812116793 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f837805c2a0 0x7f837805e760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f838805f4e0 tx=0x7f837c009040).stop 2021-11-20T05:34:30.676 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.674+0000 7f838ed18700 1 -- 172.21.15.58:0/812116793 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83880f6b80 msgr2=0x7f838810bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:30.676 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.674+0000 7f838ed18700 1 --2- 172.21.15.58:0/812116793 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83880f6b80 0x7f838810bce0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f8374007a70 tx=0x7f8374009b30).stop 2021-11-20T05:34:30.676 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.674+0000 7f838ed18700 1 -- 172.21.15.58:0/812116793 shutdown_connections 2021-11-20T05:34:30.677 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.674+0000 7f838ed18700 1 --2- 172.21.15.58:0/812116793 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f837805c2a0 0x7f837805e760 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:30.677 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.674+0000 7f838ed18700 1 --2- 172.21.15.58:0/812116793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f83880f5880 0x7f83880f5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:30.678 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.674+0000 7f838ed18700 1 --2- 172.21.15.58:0/812116793 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f83880f6b80 0x7f838810bce0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:30.678 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.674+0000 7f838ed18700 1 -- 172.21.15.58:0/812116793 >> 172.21.15.58:0/812116793 conn(0x7f83880f0470 msgr2=0x7f83880fa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:30.678 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.674+0000 7f838ed18700 1 -- 172.21.15.58:0/812116793 shutdown_connections 2021-11-20T05:34:30.679 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.674+0000 7f838ed18700 1 -- 172.21.15.58:0/812116793 wait complete. 2021-11-20T05:34:30.686 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-11-20T05:34:30.687 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:34:30.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:30 smithi187 conmon[36358]: audit 2021-11-20T05:34:29.751005+0000 mon.smithi058 (mon.0) 535 : audit [DBG] from='client.? 172.21.15.58:0/603748962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T05:34:30.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:30 smithi187 conmon[36358]: 2021-11-20T05:34:30.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:30 smithi187 conmon[36358]: audit 2021-11-20T05:34:30.056065+0000 mon.smithi058 (mon.0) 536 : audit [DBG] from='client.? 172.21.15.58:0/3197259930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T05:34:30.807 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.805+0000 7f0d71eee700 1 -- 172.21.15.58:0/740213574 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d6c0f6810 msgr2=0x7f0d6c0f6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:30.808 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.805+0000 7f0d71eee700 1 --2- 172.21.15.58:0/740213574 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d6c0f6810 0x7f0d6c0f6c70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f0d60004660 tx=0x7f0d60009b30).stop 2021-11-20T05:34:30.808 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.806+0000 7f0d71eee700 1 -- 172.21.15.58:0/740213574 shutdown_connections 2021-11-20T05:34:30.808 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.806+0000 7f0d71eee700 1 --2- 172.21.15.58:0/740213574 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d6c0f6810 0x7f0d6c0f6c70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:30.808 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.806+0000 7f0d71eee700 1 --2- 172.21.15.58:0/740213574 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d6c0f5570 0x7f0d6c0f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:30.808 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.806+0000 7f0d71eee700 1 -- 172.21.15.58:0/740213574 >> 172.21.15.58:0/740213574 conn(0x7f0d6c0f0cb0 msgr2=0x7f0d6c0f30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:30.809 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.806+0000 7f0d71eee700 1 -- 172.21.15.58:0/740213574 shutdown_connections 2021-11-20T05:34:30.809 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.806+0000 7f0d71eee700 1 -- 172.21.15.58:0/740213574 wait complete. 2021-11-20T05:34:30.809 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.806+0000 7f0d71eee700 1 Processor -- start 2021-11-20T05:34:30.809 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.806+0000 7f0d71eee700 1 -- start start 2021-11-20T05:34:30.810 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.806+0000 7f0d71eee700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d6c0f5570 0x7f0d6c060550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:30.810 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.806+0000 7f0d71eee700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d6c0f6810 0x7f0d6c10b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:30.810 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.806+0000 7f0d71eee700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c10bee0 con 0x7f0d6c0f5570 2021-11-20T05:34:30.810 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.806+0000 7f0d71eee700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c10c020 con 0x7f0d6c0f6810 2021-11-20T05:34:30.811 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.807+0000 7f0d6b7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d6c0f5570 0x7f0d6c060550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:30.811 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.807+0000 7f0d6b7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d6c0f5570 0x7f0d6c060550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40800/0 (socket says 172.21.15.58:40800) 2021-11-20T05:34:30.811 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.807+0000 7f0d6b7fe700 1 -- 172.21.15.58:0/3214638200 learned_addr learned my addr 172.21.15.58:0/3214638200 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:30.811 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.807+0000 7f0d6affd700 1 --2- 172.21.15.58:0/3214638200 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d6c0f6810 0x7f0d6c10b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:30.811 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.807+0000 7f0d6b7fe700 1 -- 172.21.15.58:0/3214638200 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d6c0f6810 msgr2=0x7f0d6c10b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:30.812 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.807+0000 7f0d6b7fe700 1 --2- 172.21.15.58:0/3214638200 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d6c0f6810 0x7f0d6c10b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:30.812 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.807+0000 7f0d6b7fe700 1 -- 172.21.15.58:0/3214638200 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d60005040 con 0x7f0d6c0f5570 2021-11-20T05:34:30.812 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.807+0000 7f0d6affd700 1 --2- 172.21.15.58:0/3214638200 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d6c0f6810 0x7f0d6c10b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:34:30.812 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.807+0000 7f0d6b7fe700 1 --2- 172.21.15.58:0/3214638200 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d6c0f5570 0x7f0d6c060550 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f0d5c00c560 tx=0x7f0d5c007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:30.813 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.807+0000 7f0d53fff700 1 -- 172.21.15.58:0/3214638200 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d5c00f650 con 0x7f0d6c0f5570 2021-11-20T05:34:30.814 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.807+0000 7f0d53fff700 1 -- 172.21.15.58:0/3214638200 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d5c00fc60 con 0x7f0d6c0f5570 2021-11-20T05:34:30.814 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.807+0000 7f0d53fff700 1 -- 172.21.15.58:0/3214638200 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d5c012d20 con 0x7f0d6c0f5570 2021-11-20T05:34:30.814 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.807+0000 7f0d71eee700 1 -- 172.21.15.58:0/3214638200 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6c10c280 con 0x7f0d6c0f5570 2021-11-20T05:34:30.814 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.807+0000 7f0d71eee700 1 -- 172.21.15.58:0/3214638200 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6c10c840 con 0x7f0d6c0f5570 2021-11-20T05:34:30.815 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.808+0000 7f0d71eee700 1 -- 172.21.15.58:0/3214638200 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d6c027a00 con 0x7f0d6c0f5570 2021-11-20T05:34:30.816 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.809+0000 7f0d53fff700 1 -- 172.21.15.58:0/3214638200 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0d5c019030 con 0x7f0d6c0f5570 2021-11-20T05:34:30.816 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.809+0000 7f0d53fff700 1 --2- 172.21.15.58:0/3214638200 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0d5405c2a0 0x7f0d5405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:30.816 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.809+0000 7f0d53fff700 1 -- 172.21.15.58:0/3214638200 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f0d5c075ae0 con 0x7f0d6c0f5570 2021-11-20T05:34:30.817 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.809+0000 7f0d6affd700 1 --2- 172.21.15.58:0/3214638200 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0d5405c2a0 0x7f0d5405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:30.817 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.809+0000 7f0d6affd700 1 --2- 172.21.15.58:0/3214638200 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0d5405c2a0 0x7f0d5405e760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f0d60009d60 tx=0x7f0d60009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:30.817 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:30.813+0000 7f0d53fff700 1 -- 172.21.15.58:0/3214638200 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d5c0462b0 con 0x7f0d6c0f5570 2021-11-20T05:34:31.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.013+0000 7f0d71eee700 1 -- 172.21.15.58:0/3214638200 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d6c0fdd00 con 0x7f0d6c0f5570 2021-11-20T05:34:31.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.013+0000 7f0d53fff700 1 -- 172.21.15.58:0/3214638200 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+2946 (secure 0 0 0) 0x7f0d5c017020 con 0x7f0d6c0f5570 2021-11-20T05:34:31.014 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:34:31.014 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24259,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2274008158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2274008158},{"type":"v1","addr":"172.21.15.187:6825","nonce":2274008158}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":9},{"gid":24271,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2997529975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2997529975},{"type":"v1","addr":"172.21.15.187:6827","nonce":2997529975}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:34:15.893368+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14428,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14428":{"gid":14428,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.58:6827/3420124164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3420124164},{"type":"v1","addr":"172.21.15.58:6827","nonce":3420124164}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:34:31.016 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.015+0000 7f0d71eee700 1 -- 172.21.15.58:0/3214638200 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0d5405c2a0 msgr2=0x7f0d5405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:31.016 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.015+0000 7f0d71eee700 1 --2- 172.21.15.58:0/3214638200 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0d5405c2a0 0x7f0d5405e760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f0d60009d60 tx=0x7f0d60009b30).stop 2021-11-20T05:34:31.016 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.015+0000 7f0d71eee700 1 -- 172.21.15.58:0/3214638200 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d6c0f5570 msgr2=0x7f0d6c060550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:31.017 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.015+0000 7f0d71eee700 1 --2- 172.21.15.58:0/3214638200 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d6c0f5570 0x7f0d6c060550 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f0d5c00c560 tx=0x7f0d5c007e10).stop 2021-11-20T05:34:31.017 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.015+0000 7f0d71eee700 1 -- 172.21.15.58:0/3214638200 shutdown_connections 2021-11-20T05:34:31.017 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.015+0000 7f0d71eee700 1 --2- 172.21.15.58:0/3214638200 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0d5405c2a0 0x7f0d5405e760 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:31.017 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.015+0000 7f0d71eee700 1 --2- 172.21.15.58:0/3214638200 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0d6c0f5570 0x7f0d6c060550 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:31.018 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.015+0000 7f0d71eee700 1 --2- 172.21.15.58:0/3214638200 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0d6c0f6810 0x7f0d6c10b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:31.018 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.015+0000 7f0d71eee700 1 -- 172.21.15.58:0/3214638200 >> 172.21.15.58:0/3214638200 conn(0x7f0d6c0f0cb0 msgr2=0x7f0d6c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:31.018 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.015+0000 7f0d71eee700 1 -- 172.21.15.58:0/3214638200 shutdown_connections 2021-11-20T05:34:31.018 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.015+0000 7f0d71eee700 1 -- 172.21.15.58:0/3214638200 wait complete. 2021-11-20T05:34:31.019 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 11 2021-11-20T05:34:31.027 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:34:15.893368+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14428, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14428': {'gid': 14428, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.58:6827/3420124164', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 3420124164}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 3420124164}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:34:31.027 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2021-11-20T05:34:31.028 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.cephfs.smithi058.rxapzd status 2021-11-20T05:34:31.149 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.147+0000 7f9ae0819700 1 -- 172.21.15.58:0/1737064835 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9adc0f59b0 msgr2=0x7f9adc0f5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:31.149 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.147+0000 7f9ae0819700 1 --2- 172.21.15.58:0/1737064835 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9adc0f59b0 0x7f9adc0f5dd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f9acc004660 tx=0x7f9acc009b30).stop 2021-11-20T05:34:31.149 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.147+0000 7f9ae0819700 1 -- 172.21.15.58:0/1737064835 shutdown_connections 2021-11-20T05:34:31.150 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.147+0000 7f9ae0819700 1 --2- 172.21.15.58:0/1737064835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9adc0f6350 0x7f9adc0f67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:31.150 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.147+0000 7f9ae0819700 1 --2- 172.21.15.58:0/1737064835 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9adc0f59b0 0x7f9adc0f5dd0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:31.150 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.147+0000 7f9ae0819700 1 -- 172.21.15.58:0/1737064835 >> 172.21.15.58:0/1737064835 conn(0x7f9adc0f02c0 msgr2=0x7f9adc0f26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:31.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.147+0000 7f9ae0819700 1 -- 172.21.15.58:0/1737064835 shutdown_connections 2021-11-20T05:34:31.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.147+0000 7f9ae0819700 1 -- 172.21.15.58:0/1737064835 wait complete. 2021-11-20T05:34:31.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.148+0000 7f9ae0819700 1 Processor -- start 2021-11-20T05:34:31.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.148+0000 7f9ae0819700 1 -- start start 2021-11-20T05:34:31.152 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.148+0000 7f9ae0819700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9adc0f59b0 0x7f9adc106da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:31.152 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.148+0000 7f9ae0819700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9adc0f6350 0x7f9adc107290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:31.152 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.148+0000 7f9ae0819700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc107890 con 0x7f9adc0f6350 2021-11-20T05:34:31.152 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.148+0000 7f9ae0819700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc1080d0 con 0x7f9adc0f59b0 2021-11-20T05:34:31.153 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.148+0000 7f9ada59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9adc0f6350 0x7f9adc107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:31.153 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.148+0000 7f9ada59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9adc0f6350 0x7f9adc107290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40812/0 (socket says 172.21.15.58:40812) 2021-11-20T05:34:31.153 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.148+0000 7f9ada59c700 1 -- 172.21.15.58:0/1648803609 learned_addr learned my addr 172.21.15.58:0/1648803609 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:31.153 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.148+0000 7f9adad9d700 1 --2- 172.21.15.58:0/1648803609 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9adc0f59b0 0x7f9adc106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:31.153 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.148+0000 7f9ada59c700 1 -- 172.21.15.58:0/1648803609 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9adc0f59b0 msgr2=0x7f9adc106da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:31.154 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.148+0000 7f9ada59c700 1 --2- 172.21.15.58:0/1648803609 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9adc0f59b0 0x7f9adc106da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:31.154 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.149+0000 7f9ada59c700 1 -- 172.21.15.58:0/1648803609 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc005040 con 0x7f9adc0f6350 2021-11-20T05:34:31.154 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.149+0000 7f9ada59c700 1 --2- 172.21.15.58:0/1648803609 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9adc0f6350 0x7f9adc107290 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f9ad0002700 tx=0x7f9ad0009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:31.154 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.149+0000 7f9ac37fe700 1 -- 172.21.15.58:0/1648803609 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ad0009320 con 0x7f9adc0f6350 2021-11-20T05:34:31.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.149+0000 7f9ae0819700 1 -- 172.21.15.58:0/1648803609 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9adc1083b0 con 0x7f9adc0f6350 2021-11-20T05:34:31.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.149+0000 7f9ac37fe700 1 -- 172.21.15.58:0/1648803609 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ad000f040 con 0x7f9adc0f6350 2021-11-20T05:34:31.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.149+0000 7f9ac37fe700 1 -- 172.21.15.58:0/1648803609 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ad0014870 con 0x7f9adc0f6350 2021-11-20T05:34:31.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.149+0000 7f9ae0819700 1 -- 172.21.15.58:0/1648803609 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9adc1088f0 con 0x7f9adc0f6350 2021-11-20T05:34:31.162 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.150+0000 7f9ac37fe700 1 -- 172.21.15.58:0/1648803609 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9ad0009480 con 0x7f9adc0f6350 2021-11-20T05:34:31.162 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.150+0000 7f9ac37fe700 1 --2- 172.21.15.58:0/1648803609 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9ac405c2b0 0x7f9ac405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:31.163 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.151+0000 7f9ac37fe700 1 -- 172.21.15.58:0/1648803609 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f9ad00761d0 con 0x7f9adc0f6350 2021-11-20T05:34:31.163 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.151+0000 7f9adad9d700 1 --2- 172.21.15.58:0/1648803609 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9ac405c2b0 0x7f9ac405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:31.163 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.151+0000 7f9adad9d700 1 --2- 172.21.15.58:0/1648803609 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9ac405c2b0 0x7f9ac405e770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f9adc107b60 tx=0x7f9acc009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:31.163 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.152+0000 7f9af2bc5b80 1 Processor -- start 2021-11-20T05:34:31.163 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.152+0000 7f9af2bc5b80 1 -- start start 2021-11-20T05:34:31.164 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.153+0000 7f9af2bc5b80 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5644ba2598a0 0x5644ba259cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:31.164 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.153+0000 7f9af2bc5b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5644ba25a1b0 0x5644ba25ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:31.164 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.153+0000 7f9af2bc5b80 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644ba1984d0 con 0x5644ba2598a0 2021-11-20T05:34:31.164 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.153+0000 7f9af2bc5b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644ba1ac3e0 con 0x5644ba25a1b0 2021-11-20T05:34:31.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.153+0000 7f9adad9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5644ba2598a0 0x5644ba259cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:31.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.153+0000 7f9adad9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5644ba2598a0 0x5644ba259cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40816/0 (socket says 172.21.15.58:40816) 2021-11-20T05:34:31.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.153+0000 7f9adad9d700 1 -- 172.21.15.58:0/2531347475 learned_addr learned my addr 172.21.15.58:0/2531347475 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:31.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.153+0000 7f9ada59c700 1 --2- 172.21.15.58:0/2531347475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5644ba25a1b0 0x5644ba25ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:31.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.153+0000 7f9adad9d700 1 -- 172.21.15.58:0/2531347475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5644ba25a1b0 msgr2=0x5644ba25ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T05:34:31.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.153+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2531347475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5644ba25a1b0 0x5644ba25ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:31.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.153+0000 7f9adad9d700 1 -- 172.21.15.58:0/2531347475 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644ba2283c0 con 0x5644ba2598a0 2021-11-20T05:34:31.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.153+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2531347475 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5644ba2598a0 0x5644ba259cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f9adc05f350 tx=0x7f9acc01d000).ready entity=mon.0 client_cookie=85458eab1e1f12ad server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:31.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.153+0000 7f9ac0be5700 1 -- 172.21.15.58:0/2531347475 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9acc00e040 con 0x5644ba2598a0 2021-11-20T05:34:31.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.153+0000 7f9ac0be5700 1 -- 172.21.15.58:0/2531347475 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9acc019040 con 0x5644ba2598a0 2021-11-20T05:34:31.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.153+0000 7f9ac0be5700 1 -- 172.21.15.58:0/2531347475 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9acc029c80 con 0x5644ba2598a0 2021-11-20T05:34:31.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.154+0000 7f9af2bc5b80 1 -- 172.21.15.58:0/2531347475 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5644ba2598a0 msgr2=0x5644ba259cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:34:31.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.154+0000 7f9af2bc5b80 1 --2- 172.21.15.58:0/2531347475 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5644ba2598a0 0x5644ba259cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f9adc05f350 tx=0x7f9acc01d000).stop 2021-11-20T05:34:31.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.154+0000 7f9af2bc5b80 1 -- 172.21.15.58:0/2531347475 shutdown_connections 2021-11-20T05:34:31.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.154+0000 7f9af2bc5b80 1 --2- 172.21.15.58:0/2531347475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5644ba25a1b0 0x5644ba25ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:31.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.154+0000 7f9af2bc5b80 1 --2- 172.21.15.58:0/2531347475 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5644ba2598a0 0x5644ba259cc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:31.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.154+0000 7f9af2bc5b80 1 -- 172.21.15.58:0/2531347475 >> 172.21.15.58:0/2531347475 conn(0x5644ba0c71d0 msgr2=0x5644ba254e00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:34:31.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.154+0000 7f9af2bc5b80 1 -- 172.21.15.58:0/2531347475 shutdown_connections 2021-11-20T05:34:31.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.154+0000 7f9af2bc5b80 1 -- 172.21.15.58:0/2531347475 wait complete. 2021-11-20T05:34:31.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.154+0000 7f9af2bc5b80 1 -- start start 2021-11-20T05:34:31.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.154+0000 7f9af2bc5b80 1 Processor -- start 2021-11-20T05:34:31.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9af2bc5b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5644ba2598a0 0x5644ba262a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:31.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9af2bc5b80 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5644ba25a1b0 0x5644ba262ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:31.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9af2bc5b80 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644ba1ac520 con 0x5644ba25a1b0 2021-11-20T05:34:31.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9af2bc5b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644ba1acaa0 con 0x5644ba2598a0 2021-11-20T05:34:31.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9af2bc5b80 10 client.0 ll_register_callbacks cb 0x5644ba2198b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-20T05:34:31.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9ada59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5644ba25a1b0 0x5644ba262ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:31.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9ada59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5644ba25a1b0 0x5644ba262ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:40822/0 (socket says 172.21.15.58:40822) 2021-11-20T05:34:31.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9ada59c700 1 -- 172.21.15.58:0/2412268982 learned_addr learned my addr 172.21.15.58:0/2412268982 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:34:31.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5644ba2598a0 0x5644ba262a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:31.172 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9ada59c700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5644ba2598a0 msgr2=0x5644ba262a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T05:34:31.172 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9ada59c700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5644ba2598a0 0x5644ba262a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:34:31.172 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9ada59c700 1 -- 172.21.15.58:0/2412268982 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9adc0f7050 con 0x5644ba25a1b0 2021-11-20T05:34:31.172 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9ada59c700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x5644ba25a1b0 0x5644ba262ef0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f9ad00045b0 tx=0x7f9ad004f040).ready entity=mon.0 client_cookie=a8f7c5eca1e9f65f server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:34:31.173 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9ab9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.58:3300/0 2021-11-20T05:34:31.173 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9ab9ffb700 1 -- 172.21.15.58:0/2412268982 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ad0048ad0 con 0x5644ba25a1b0 2021-11-20T05:34:31.174 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9ab9ffb700 1 -- 172.21.15.58:0/2412268982 <== mon.0 v2:172.21.15.58:3300/0 2 ==== osd_map(42..42 src has 1..42) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f9ad00077b0 con 0x5644ba25a1b0 2021-11-20T05:34:31.174 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9af2bc5b80 1 -- 172.21.15.58:0/2412268982 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5644ba227ce0 con 0x5644ba25a1b0 2021-11-20T05:34:31.174 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9ab9ffb700 1 -- 172.21.15.58:0/2412268982 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_get_version_reply(handle=1 version=11) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ad0048c30 con 0x5644ba25a1b0 2021-11-20T05:34:31.175 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9af2bc5b80 10 client.14552 fetch_fsmap learned FSMap version 11 2021-11-20T05:34:31.175 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.155+0000 7f9af2bc5b80 1 -- 172.21.15.58:0/2412268982 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,fsmap=11,monmap=3+}) v3 -- 0x5644ba19b630 con 0x5644ba25a1b0 2021-11-20T05:34:31.175 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.156+0000 7f9ab9ffb700 1 -- 172.21.15.58:0/2412268982 <== mon.0 v2:172.21.15.58:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x5644ba227ce0 con 0x5644ba25a1b0 2021-11-20T05:34:31.175 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.156+0000 7f9ab9ffb700 1 -- 172.21.15.58:0/2412268982 <== mon.0 v2:172.21.15.58:3300/0 5 ==== fsmap(e 11) v1 ==== 1736+0+0 (secure 0 0 0) 0x7f9ad004b5b0 con 0x5644ba25a1b0 2021-11-20T05:34:31.175 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.156+0000 7f9af2bc5b80 10 client.14552 fetch_fsmap finished waiting for FSMap version 11 2021-11-20T05:34:31.176 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.156+0000 7f9af2bc5b80 10 client.14552 resolve_mds: resolved name 'cephfs.smithi058.rxapzd' to daemon mds.cephfs.smithi058.rxapzd 2021-11-20T05:34:31.176 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.156+0000 7f9af2bc5b80 20 client.14552 populate_metadata read hostname 'smithi058' 2021-11-20T05:34:31.176 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.156+0000 7f9af2bc5b80 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:34:31.176 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.156+0000 7f9af2bc5b80 4 client.14552 mds_command: new command op to 14428 tid=0[{"prefix": "get_command_descriptions"}] 2021-11-20T05:34:31.177 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.156+0000 7f9af2bc5b80 1 -- 172.21.15.58:0/2412268982 --> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5644ba1a2df0 con 0x5644ba26b770 2021-11-20T05:34:31.177 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.156+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.58:6826/3420124164 2021-11-20T05:34:31.177 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.156+0000 7f9ab9ffb700 1 client.14552 ms_handle_refused on v2:172.21.15.58:6826/3420124164 2021-11-20T05:34:31.177 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.156+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T05:34:31.356 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.357+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.58:6826/3420124164 2021-11-20T05:34:31.356 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.357+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T05:34:31.356 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.357+0000 7f9ab9ffb700 1 client.14552 ms_handle_refused on v2:172.21.15.58:6826/3420124164 2021-11-20T05:34:31.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:31 smithi187 conmon[36358]: cluster 2021-11-20T05:34:30.214729+0000 mgr.smithi058.zagdlk (mgr.14164) 204 : cluster 2021-11-20T05:34:31.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:31 smithi187 conmon[36358]: [DBG] pgmap v166: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 2 op/s 2021-11-20T05:34:31.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:31 smithi187 conmon[36358]: audit 2021-11-20T05:34:30.371295+0000 mon.smithi058 (mon.0) 537 : audit [DBG] from='client.? 172.21.15.58:0/1071919579' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T05:34:31.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:31 smithi187 conmon[36358]: audit 2021- 2021-11-20T05:34:31.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:31 smithi187 conmon[36358]: 11-20T05:34:30.673214+0000 mon.smithi058 (mon.0) 538 : audit [DBG] from='client.? 172.21.15.58:0/812116793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T05:34:31.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:31 smithi187 conmon[36358]: audit 2021-11-20T05:34: 2021-11-20T05:34:31.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:31 smithi187 conmon[36358]: 31.014347+0000 mon.smithi058 (mon.0) 539 : audit 2021-11-20T05:34:31.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:31 smithi187 conmon[36358]: [DBG] from='client.? 172.21.15.58:0/3214638200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:34:31.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:31 smithi058 conmon[33537]: cluster 2021-11-20T05:34:30.214729+0000 mgr.smithi058.zagdlk (mgr.14164) 204 : cluster [DBG] pgmap v166: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 2 op/s 2021-11-20T05:34:31.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:31 smithi058 conmon[33537]: audit 2021-11-20T05:34:30.371295+0000 mon.smithi058 (mon.0) 537 : audit [DBG] from='client.? 172.21.15.58:0/1071919579' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T05:34:31.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:31 smithi058 conmon[33537]: audit 2021-11-20T05:34:30.673214+0000 mon.smithi058 (mon.0) 538 : audit [DBG] from='client.? 172.21.15.58:0/812116793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T05:34:31.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:31 smithi058 conmon[33537]: audit 2021-11-20T05:34:31.014347+0000 2021-11-20T05:34:31.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:31 smithi058 conmon[33537]: mon.smithi058 (mon.0) 2021-11-20T05:34:31.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:31 smithi058 conmon[33537]: 539 : audit [DBG] from='client.? 172.21.15.58:0/3214638200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:34:31.757 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.758+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.58:6826/3420124164 2021-11-20T05:34:31.757 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.758+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-20T05:34:31.758 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:31.758+0000 7f9ab9ffb700 1 client.14552 ms_handle_refused on v2:172.21.15.58:6826/3420124164 2021-11-20T05:34:32.558 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:32.559+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.58:6826/3420124164 2021-11-20T05:34:32.559 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:32.559+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-20T05:34:32.559 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:32.559+0000 7f9ab9ffb700 1 client.14552 ms_handle_refused on v2:172.21.15.58:6826/3420124164 2021-11-20T05:34:33.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:33 smithi058 conmon[33537]: cluster 2021-11-20T05:34:32.215381+0000 mgr.smithi058.zagdlk (mgr.14164) 205 : cluster 2021-11-20T05:34:33.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:33 smithi058 conmon[33537]: [DBG] pgmap v167: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2021-11-20T05:34:33.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:33 smithi187 conmon[36358]: cluster 2021-11-20T05:34:32.215381+0000 mgr.smithi058.zagdlk (mgr.14164) 205 : cluster [DBG] pgmap v167: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2021-11-20T05:34:34.160 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:34.160+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.58:6826/3420124164 2021-11-20T05:34:34.160 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:34.160+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2021-11-20T05:34:34.161 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:34.160+0000 7f9ab9ffb700 1 client.14552 ms_handle_refused on v2:172.21.15.58:6826/3420124164 2021-11-20T05:34:35.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:35 smithi187 conmon[36358]: cluster 2021-11-20T05:34:34.216018+0000 mgr.smithi058.zagdlk (mgr.14164) 206 : cluster [DBG] pgmap v168: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2021-11-20T05:34:35.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:35 smithi058 conmon[33537]: cluster 2021-11-20T05:34:34.216018+0000 mgr.smithi058.zagdlk (mgr.14164) 206 2021-11-20T05:34:35.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:35 smithi058 conmon[33537]: : cluster [DBG] pgmap v168: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2021-11-20T05:34:37.363 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:37.364+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.58:6826/3420124164 2021-11-20T05:34:37.364 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:37.364+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 6.400000 2021-11-20T05:34:37.364 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:37.364+0000 7f9ab9ffb700 1 client.14552 ms_handle_refused on v2:172.21.15.58:6826/3420124164 2021-11-20T05:34:37.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:37 smithi187 conmon[36358]: cluster 2021-11-20T05:34:36.216516+0000 mgr.smithi058.zagdlk (mgr.14164) 207 : cluster [DBG] pgmap v169: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2021-11-20T05:34:37.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:37 smithi187 conmon[36358]: 2021-11-20T05:34:37.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:37 smithi058 conmon[33537]: cluster 2021-11-20T05:34:36.216516+0000 mgr.smithi058.zagdlk (mgr.14164) 207 2021-11-20T05:34:37.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:37 smithi058 conmon[33537]: : cluster [DBG] pgmap v169: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2021-11-20T05:34:39.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:39 smithi187 conmon[36358]: cluster 2021-11-20T05:34:38.217124+0000 mgr.smithi058.zagdlk (mgr.14164) 208 : cluster [DBG] pgmap v170: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2021-11-20T05:34:39.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:39 smithi187 conmon[36358]: 2021-11-20T05:34:39.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:39 smithi058 conmon[33537]: cluster 2021-11-20T05:34:38.217124+0000 mgr.smithi058.zagdlk (mgr.14164) 208 2021-11-20T05:34:39.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:39 smithi058 conmon[33537]: : cluster [DBG] pgmap v170: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2021-11-20T05:34:41.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:41 smithi187 conmon[36358]: cluster 2021-11-20T05:34:40.217807+0000 mgr.smithi058.zagdlk (mgr.14164) 209 : cluster [DBG] pgmap v171: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2021-11-20T05:34:41.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:41 smithi058 conmon[33537]: cluster 2021-11-20T05:34:40.217807+0000 mgr.smithi058.zagdlk (mgr.14164) 209 2021-11-20T05:34:41.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:41 smithi058 conmon[33537]: : cluster [DBG] pgmap v171: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2021-11-20T05:34:43.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:43 smithi187 conmon[36358]: cluster 2021-11-20T05:34:42.218289+0000 mgr.smithi058.zagdlk (mgr.14164) 210 : cluster [DBG] pgmap v172: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:43.579 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:43 smithi187 conmon[36358]: 2021-11-20T05:34:43.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:43 smithi058 conmon[33537]: cluster 2021-11-20T05:34:42.218289+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:34:43.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:43 smithi058 conmon[33537]: ) 210 : cluster [DBG] pgmap v172: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:43.771 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:43.771+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:43.771 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:43.771+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T05:34:43.771 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:43.771+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T05:34:43.772 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:43.771+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T05:34:43.772 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:43.771+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 12.800000 2021-11-20T05:34:44.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:44 smithi187 conmon[36358]: cluster 2021-11-20T05:34:44.224015+0000 mon.smithi058 (mon.0) 540 : cluster [WRN] Replacing daemon mds.cephfs.smithi058.rxapzd as rank 0 with standby daemon mds.cephfs.smithi187.wxduhu 2021-11-20T05:34:44.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:44 smithi187 conmon[36358]: cluster 2021-11-20T05:34:44.224125+0000 mon.smithi058 (mon.0) 541 : cluster [INF] MDS daemon mds.cephfs.smithi058.rxapzd is removed because it is dead or otherwise unavailable. 2021-11-20T05:34:44.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:44 smithi187 conmon[36358]: cluster 2021-11-20T05:34:44.224713+0000 mon.smithi058 (mon.0) 542 : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2021-11-20T05:34:44.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:44 smithi187 conmon[36358]: cluster 2021-11-20T05:34:44.228572+0000 mon.smithi058 (mon 2021-11-20T05:34:44.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:44 smithi187 conmon[36358]: .0) 543 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2021-11-20T05:34:44.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:44 smithi187 conmon[36358]: cluster 2021-11-20T05:34:44.234267+0000 mon.smithi058 (mon.0) 544 : cluster [DBG] 2021-11-20T05:34:44.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:44 smithi187 conmon[36358]: fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:34:44.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:44 smithi058 conmon[33537]: cluster 2021-11-20T05:34:44.224015+0000 mon.smithi058 (mon.0) 540 : 2021-11-20T05:34:44.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:44 smithi058 conmon[33537]: cluster [WRN] Replacing daemon mds.cephfs.smithi058.rxapzd as rank 0 with standby daemon mds.cephfs.smithi187.wxduhu 2021-11-20T05:34:44.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:44 smithi058 conmon[33537]: cluster 2021-11-20T05:34:44.224125+0000 mon.smithi058 (mon.0) 541 : cluster [INF] MDS daemon mds.cephfs.smithi058.rxapzd is removed because it is dead or otherwise unavailable. 2021-11-20T05:34:44.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:44 smithi058 conmon[33537]: cluster 2021-11-20T05:34:44.224713+0000 mon.smithi058 (mon.0) 542 : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2021-11-20T05:34:44.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:44 smithi058 conmon[33537]: 2021-11-20T05:34:44.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:44 smithi058 conmon[33537]: cluster 2021-11-20T05:34:44.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:44 smithi058 conmon[33537]: 2021-11-20T05:34:44.228572+0000 mon.smithi058 (mon.0) 543 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2021-11-20T05:34:44.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:44 smithi058 conmon[33537]: 2021-11-20T05:34:44.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:44 smithi058 conmon[33537]: cluster 2021-11-20T05: 2021-11-20T05:34:44.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:44 smithi058 conmon[33537]: 34:44.234267+0000 mon.smithi058 (mon.0) 544 : 2021-11-20T05:34:44.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:44 smithi058 conmon[33537]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:34:45.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:45 smithi187 conmon[36358]: cluster 2021-11-20T05:34:44.218806+0000 mgr.smithi058.zagdlk (mgr.14164) 211 : cluster [DBG] pgmap v173: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:45.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:45 smithi187 conmon[36358]: cluster 2021-11-20T05:34:45.236278+0000 mon.smithi058 (mon.0) 545 : cluster [DBG] mds.? [v2:172.21.15.187:6824/2274008158,v1:172.21.15.187:6825/2274008158] up:resolve 2021-11-20T05:34:45.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:45 smithi187 conmon[36358]: cluster 2021-11-20T05:34:45.236348+0000 mon.smithi058 (mon.0) 546 : cluster [DBG] 2021-11-20T05:34:45.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:45 smithi187 conmon[36358]: mds.? [v2:172.21.15.58:6826/1536802493,v1:172.21.15.58:6827/1536802493] up:boot 2021-11-20T05:34:45.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:45 smithi187 conmon[36358]: cluster 2021-11-20T05:34:45.236420+0000 mon.smithi058 (mon.0) 547 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:34:45.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:45 smithi187 conmon[36358]: 2021-11-20T05:34:45.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:45 smithi187 conmon[36358]: audit 2021-11-20T05:34:45.236580+0000 mon.smithi058 (mon 2021-11-20T05:34:45.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:45 smithi187 conmon[36358]: .0) 548 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi058.rxapzd"}]: dispatch 2021-11-20T05:34:45.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:45 smithi058 conmon[33537]: cluster 2021-11-20T05:34:44.218806+0000 mgr.smithi058.zagdlk (mgr.14164) 211 : cluster [DBG] pgmap v173: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:45.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:45 smithi058 conmon[33537]: cluster 2021-11-20T05:34:45.236278+0000 mon.smithi058 (mon.0) 545 : cluster [DBG] mds.? [v2:172.21.15.187:6824/2274008158,v1:172.21.15.187:6825/2274008158] up:resolve 2021-11-20T05:34:45.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:45 smithi058 conmon[33537]: cluster 2021-11-20T05:34:45.236348+0000 mon.smithi058 (mon. 2021-11-20T05:34:45.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:45 smithi058 conmon[33537]: 0) 546 : cluster [DBG] mds.? [v2:172.21.15.58:6826/1536802493,v1:172.21.15.58:6827/1536802493] up:boot 2021-11-20T05:34:45.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:45 smithi058 conmon[33537]: cluster 2021-11-20T05:34:45.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:45 smithi058 conmon[33537]: 2021-11-20T05:34:45.236420 2021-11-20T05:34:45.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:45 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 547 : cluster 2021-11-20T05:34:45.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:45 smithi058 conmon[33537]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:34:45.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:45 smithi058 conmon[33537]: audit 2021-11-20T05:34:45.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:45 smithi058 conmon[33537]: 2021-11 2021-11-20T05:34:45.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:45 smithi058 conmon[33537]: -20T05:34:45 2021-11-20T05:34:45.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:45 smithi058 conmon[33537]: .236580+0000 mon.smithi058 (mon.0) 548 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi058.rxapzd"}]: dispatch 2021-11-20T05:34:47.682 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:47 smithi187 conmon[36358]: cluster 2021-11-20T05:34:46.219477+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:34:47.683 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:47 smithi187 conmon[36358]: 212 : cluster [DBG] pgmap v175: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:47.683 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:47 smithi187 conmon[36358]: cluster 2021-11-20T05:34:46.237740+0000 mon.smithi058 (mon.0) 549 : cluster [DBG] mds.? [v2:172.21.15.187:6824/2274008158,v1:172.21.15.187:6825/2274008158] up:reconnect 2021-11-20T05:34:47.683 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:47 smithi187 conmon[36358]: 2021-11-20T05:34:47.683 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:47 smithi187 conmon[36358]: cluster 2021-11-20T05:34:46.237831+0000 mon.smithi058 ( 2021-11-20T05:34:47.684 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:47 smithi187 conmon[36358]: mon.0) 550 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:34:47.684 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:47 smithi187 conmon[36358]: cluster 2021-11-20T05:34:46 2021-11-20T05:34:47.684 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:47 smithi187 conmon[36358]: .257879+0000 mds.cephfs.smithi187.wxduhu (mds.0) 1 : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0219997 2021-11-20T05:34:47.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:47 smithi058 conmon[33537]: cluster 2021-11-20T05:34:46.219477+0000 mgr.smithi058.zagdlk (mgr.14164) 212 : cluster 2021-11-20T05:34:47.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:47 smithi058 conmon[33537]: [DBG] pgmap v175: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:34:47.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:47 smithi058 conmon[33537]: cluster 2021-11-20T05:34:46.237740+0000 mon.smithi058 (mon.0) 549 : cluster [DBG] mds.? [v2:172.21.15.187:6824/2274008158,v1:172.21.15.187:6825/2274008158] up:reconnect 2021-11-20T05:34:47.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:47 smithi058 conmon[33537]: cluster 2021-11-20T05:34:46.237831+0000 mon.smithi058 (mon.0) 550 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:34:47.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:47 smithi058 conmon[33537]: cluster 2021-11-20T05:34:46.257879+0000 mds.cephfs.smithi187.wxduhu (mds.0) 2021-11-20T05:34:47.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:47 smithi058 conmon[33537]: 1 : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0219997 2021-11-20T05:34:48.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:48 smithi058 conmon[33537]: cluster 2021-11-20T05:34:47.242404+0000 mon.smithi058 (mon.0) 551 : 2021-11-20T05:34:48.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:48 smithi058 conmon[33537]: cluster [DBG] mds.? [v2:172.21.15.187:6824/2274008158,v1:172.21.15.187:6825/2274008158] up:rejoin 2021-11-20T05:34:48.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:48 smithi058 conmon[33537]: cluster 2021-11-20T05:34:47.242612+0000 mon.smithi058 (mon.0) 552 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:34:48.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:48 smithi187 conmon[36358]: cluster 2021-11-20T05:34:47.242404+0000 mon.smithi058 (mon.0) 551 : cluster [DBG] mds.? [v2:172.21.15.187:6824/2274008158,v1:172.21.15.187:6825/2274008158] up:rejoin 2021-11-20T05:34:48.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:48 smithi187 conmon[36358]: cluster 2021-11-20T05:34:47.242612+0000 mon.smithi058 (mon.0) 552 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:34:49.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:49 smithi187 conmon[36358]: cluster 2021-11-20T05:34:48.219989+0000 mgr.smithi058.zagdlk (mgr.14164) 213 : 2021-11-20T05:34:49.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:49 smithi187 conmon[36358]: cluster [DBG] pgmap v176: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2021-11-20T05:34:49.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:49 smithi058 conmon[33537]: cluster 2021-11-20T05:34 2021-11-20T05:34:49.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:49 smithi058 conmon[33537]: :48.219989+0000 mgr.smithi058.zagdlk (mgr.14164) 213 : cluster [DBG] pgmap v176: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2021-11-20T05:34:50.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:50 smithi187 conmon[36358]: cluster 2021-11-20T05:34:49.253785+0000 mon.smithi058 (mon.0) 553 : cluster [DBG] mds.? [v2:172.21.15.58:6826/1536802493,v1:172.21.15.58:6827/1536802493] up:standby 2021-11-20T05:34:50.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:50 smithi187 conmon[36358]: cluster 2021-11-20T05:34:49.253891+0000 mon.smithi058 (mon.0) 554 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:34:50.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:50 smithi058 conmon[33537]: cluster 2021-11-20T05:34:49.253785+0000 mon.smithi058 (mon. 2021-11-20T05:34:50.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:50 smithi058 conmon[33537]: 0) 553 : cluster [DBG] mds.? [v2:172.21.15.58:6826/1536802493,v1:172.21.15.58:6827/1536802493] up:standby 2021-11-20T05:34:50.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:50 smithi058 conmon[33537]: cluster 2021-11-20T05:34:49.253891+0000 mon.smithi058 (mon.0) 554 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:34:51.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:51 smithi187 conmon[36358]: cluster 2021-11-20T05:34:50.220526+0000 mgr.smithi058.zagdlk (mgr.14164) 214 : cluster [DBG] pgmap v177: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:34:51.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:51 smithi058 conmon[33537]: cluster 2021-11-20T05:34:50. 2021-11-20T05:34:51.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:51 smithi058 conmon[33537]: 220526+0000 mgr.smithi058.zagdlk (mgr.14164) 214 : cluster [DBG] pgmap v177: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:34:53.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:53 smithi187 conmon[36358]: cluster 2021-11-20T05:34:52.223727+0000 mgr.smithi058.zagdlk (mgr.14164) 215 : cluster [DBG] pgmap v178: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:34:53.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:53 smithi058 conmon[33537]: cluster 2021-11-20T05:34:52.223727+0000 mgr.smithi058.zagdlk (mgr.14164) 215 : cluster [DBG] pgmap v178: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:34:53.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:53 smithi058 conmon[33537]: 2021-11-20T05:34:55.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:55 smithi187 conmon[36358]: cluster 2021-11-20T05:34:54.225167+0000 mgr.smithi058.zagdlk (mgr.14164) 216 : cluster [DBG] pgmap v179: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:34:55.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:55 smithi058 conmon[33537]: cluster 2021-11-20T05:34:54.225167+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:34:55.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:55 smithi058 conmon[33537]: mgr.14164) 216 : cluster [DBG] pgmap v179: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:34:56.584 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:56.584+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:34:56.585 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:56.584+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T05:34:56.585 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:56.584+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T05:34:56.585 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:56.584+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T05:34:56.585 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:34:56.584+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T05:34:57.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:57 smithi187 conmon[36358]: cluster 2021-11-20T05:34:56.225624+0000 mgr.smithi058.zagdlk (mgr.14164) 217 : cluster [DBG] pgmap v180: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:34:57.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:57 smithi058 conmon[33537]: cluster 2021-11-20T05:34:56.225624+0000 mgr.smithi058.zagdlk (mgr.14164) 217 : cluster 2021-11-20T05:34:57.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:57 smithi058 conmon[33537]: [DBG] pgmap v180: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:34:58.275 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:58 smithi187 conmon[36358]: cluster 2021-11-20T05:34:58.204811+0000 mon.smithi058 (mon.0 2021-11-20T05:34:58.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:58 smithi187 conmon[36358]: ) 555 : cluster [INF] daemon mds.cephfs.smithi187.wxduhu restarted 2021-11-20T05:34:58.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:58 smithi187 conmon[36358]: cluster 2021-11-20T05:34:58.209446+0000 mon.smithi058 (mon.0) 556 2021-11-20T05:34:58.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:58 smithi187 conmon[36358]: : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2021-11-20T05:34:58.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:58 smithi058 conmon[33537]: cluster 2021-11-20T05:34:58.204811+0000 mon.smithi058 (mon.0) 555 : cluster 2021-11-20T05:34:58.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:58 smithi058 conmon[33537]: [INF] daemon mds.cephfs.smithi187.wxduhu restarted 2021-11-20T05:34:58.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:58 smithi058 conmon[33537]: cluster 2021-11-20T05:34:58.209446+0000 mon.smithi058 (mon.0) 556 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2021-11-20T05:34:59.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:59 smithi187 conmon[36358]: cluster 2021-11-20T05:34:58.225987+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:34:59.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:59 smithi187 conmon[36358]: ) 218 : cluster [DBG] pgmap v182: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T05:34:59.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:59 smithi187 conmon[36358]: cluster 2021-11-20T05:34:58.280876+0000 mon.smithi058 (mon.0) 557 : cluster [DBG] mds.? [v2:172.21.15.187:6824/4174217961,v1:172.21.15.187:6825/4174217961] up:boot 2021-11-20T05:34:59.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:59 smithi187 conmon[36358]: cluster 2021-11-20T05:34:58.280968+0000 mon.smithi058 (mon.0) 2021-11-20T05:34:59.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:59 smithi187 conmon[36358]: 558 : cluster [INF] Standby daemon mds.cephfs.smithi058.rxapzd assigned to filesystem cephfs as rank 0 2021-11-20T05:34:59.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:59 smithi187 conmon[36358]: cluster 2021-11-20T05: 2021-11-20T05:34:59.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:59 smithi187 conmon[36358]: 34:58.281277+0000 mon.smithi058 (mon.0) 559 : cluster 2021-11-20T05:34:59.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:59 smithi187 conmon[36358]: [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 3 up:standby, 1 failed 2021-11-20T05:34:59.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:59 smithi187 conmon[36358]: audit 2021-11-20T05 2021-11-20T05:34:59.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:59 smithi187 conmon[36358]: :34:58.281420+0000 mon.smithi058 (mon.0) 560 2021-11-20T05:34:59.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:59 smithi187 conmon[36358]: : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.wxduhu"}]: dispatch 2021-11-20T05:34:59.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:59 smithi187 conmon[36358]: cluster 2021-11 2021-11-20T05:34:59.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:59 smithi187 conmon[36358]: -20T05:34:58.284018+0000 mon.smithi058 (mon.0 2021-11-20T05:34:59.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:34:59 smithi187 conmon[36358]: ) 561 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:34:59.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:59 smithi058 conmon[33537]: cluster 2021-11-20T05:34:58.225987+0000 mgr.smithi058.zagdlk (mgr.14164) 218 : cluster [DBG] pgmap v182: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T05:34:59.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:59 smithi058 conmon[33537]: cluster 2021-11-20T05:34:58.280876+0000 mon.smithi058 (mon.0) 557 : cluster [DBG] mds.? [v2:172.21.15.187:6824/4174217961,v1:172.21.15.187:6825/4174217961] up:boot 2021-11-20T05:34:59.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:59 smithi058 conmon[33537]: cluster 2021-11-20T05:34: 2021-11-20T05:34:59.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:59 smithi058 conmon[33537]: 58.280968+0000 mon.smithi058 (mon.0) 558 2021-11-20T05:34:59.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:59 smithi058 conmon[33537]: : cluster [INF] Standby daemon mds.cephfs.smithi058.rxapzd assigned to filesystem cephfs as rank 0 2021-11-20T05:34:59.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:59 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:34:59.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:59 smithi058 conmon[33537]: -11-20T05:34:58 2021-11-20T05:34:59.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:59 smithi058 conmon[33537]: .281277+0000 mon.smithi058 (mon.0) 559 : 2021-11-20T05:34:59.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:59 smithi058 conmon[33537]: cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 3 up:standby, 1 failed 2021-11-20T05:34:59.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:59 smithi058 conmon[33537]: audit 2021 2021-11-20T05:34:59.724 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:59 smithi058 conmon[33537]: -11-20T05:34:58 2021-11-20T05:34:59.724 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:59 smithi058 conmon[33537]: .281420+0000 mon.smithi058 (mon.0) 560 2021-11-20T05:34:59.724 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:59 smithi058 conmon[33537]: : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.wxduhu"}]: dispatch 2021-11-20T05:34:59.724 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:59 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:34:59.725 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:59 smithi058 conmon[33537]: -11-20T05:34:58 2021-11-20T05:34:59.725 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:59 smithi058 conmon[33537]: .284018+0000 mon.smithi058 (mon.0 2021-11-20T05:34:59.725 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:34:59 smithi058 conmon[33537]: ) 561 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:00.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:00 smithi187 conmon[36358]: cluster 2021-11-20T05:34:59.286179+0000 mon.smithi058 (mon.0) 562 : cluster [DBG] mds.? [v2:172.21.15.58:6826/1536802493,v1:172.21.15.58:6827/1536802493] up:resolve 2021-11-20T05:35:00.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:00 smithi187 conmon[36358]: cluster 2021-11-20T05:34:59.286234+0000 mon.smithi058 (mon.0) 563 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:00.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:00 smithi058 conmon[33537]: cluster 2021-11-20T05:34:59.286179+0000 mon.smithi058 (mon.0) 562 : cluster [DBG] mds.? [v2:172.21.15.58:6826/1536802493,v1:172.21.15.58:6827/1536802493] up:resolve 2021-11-20T05:35:00.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:00 smithi058 conmon[33537]: cluster 2021-11-20T05:34:59.286234+0000 mon.smithi058 (mon.0) 563 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:01.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:01 smithi187 conmon[36358]: cluster 2021-11-20T05:35:00.226486+0000 mgr.smithi058.zagdlk (mgr.14164) 219 : 2021-11-20T05:35:01.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:01 smithi187 conmon[36358]: cluster [DBG] pgmap v183: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T05:35:01.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:01 smithi187 conmon[36358]: cluster 2021-11-20T05:35:00.291528+0000 mon.smithi058 (mon.0) 564 : cluster [DBG] mds.? [v2:172.21.15.58:6826/1536802493,v1:172.21.15.58:6827/1536802493] up:reconnect 2021-11-20T05:35:01.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:01 smithi187 conmon[36358]: cluster 2021-11-20T05:35:00.291624+0000 2021-11-20T05:35:01.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:01 smithi187 conmon[36358]: mon.smithi058 (mon.0) 565 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:01.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:01 smithi187 conmon[36358]: cluster 2021- 2021-11-20T05:35:01.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:01 smithi187 conmon[36358]: 11-20T05:35:00. 2021-11-20T05:35:01.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:01 smithi187 conmon[36358]: 313456+0000 mds.cephfs.smithi058.rxapzd (mds.0) 1 : cluster 2021-11-20T05:35:01.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:01 smithi187 conmon[36358]: [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0229996 2021-11-20T05:35:01.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:01 smithi187 conmon[36358]: audit 2021-11-20 2021-11-20T05:35:01.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:01 smithi187 conmon[36358]: T05:35:00.382744+0000 mon.smithi058 ( 2021-11-20T05:35:01.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:01 smithi187 conmon[36358]: mon.0) 566 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:35:01.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:01 smithi187 conmon[36358]: audit 2021 2021-11-20T05:35:01.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:01 smithi187 conmon[36358]: -11-20T05:35:00. 2021-11-20T05:35:01.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:01 smithi187 conmon[36358]: 382989+0000 mon.smithi058 (mon.0) 567 : audit [INF] 2021-11-20T05:35:01.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:01 smithi187 conmon[36358]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:35:01.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:01 smithi058 conmon[33537]: cluster 2021-11-20T05:35:00.226486+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:35:01.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:01 smithi058 conmon[33537]: mgr.14164) 219 : cluster [DBG] pgmap v183: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T05:35:01.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:01 smithi058 conmon[33537]: cluster 2021-11-20T05:35:00.291528+0000 mon.smithi058 (mon.0) 564 : cluster [DBG] mds.? [v2:172.21.15.58:6826/1536802493,v1:172.21.15.58:6827/1536802493] up:reconnect 2021-11-20T05:35:01.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:01 smithi058 conmon[33537]: cluster 2021-11-20T05:35:00. 2021-11-20T05:35:01.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:01 smithi058 conmon[33537]: 291624+0000 mon.smithi058 (mon 2021-11-20T05:35:01.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:01 smithi058 conmon[33537]: .0) 565 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:01.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:01 smithi058 conmon[33537]: cluster 2021-11 2021-11-20T05:35:01.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:01 smithi058 conmon[33537]: -20T05:35:00.313456+0000 mds.cephfs.smithi058.rxapzd (mds 2021-11-20T05:35:01.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:01 smithi058 conmon[33537]: .0) 1 : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0229996 2021-11-20T05:35:01.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:01 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:35:01.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:01 smithi058 conmon[33537]: 11-20T05:35:00.382744+0000 mon.smithi058 (mon.0) 566 : 2021-11-20T05:35:01.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:01 smithi058 conmon[33537]: audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:35:01.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:01 smithi058 conmon[33537]: audit 2021-11-20T05: 2021-11-20T05:35:01.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:01 smithi058 conmon[33537]: 35:00.382989+0000 mon.smithi058 (mon.0 2021-11-20T05:35:01.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:01 smithi058 conmon[33537]: ) 567 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:35:02.465 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:02 smithi058 conmon[33537]: cluster 2021-11-20T05:35:01.297597+0000 mon.smithi058 (mon.0) 568 : cluster [DBG] mds.? [v2:172.21.15.58:6826/1536802493,v1:172.21.15.58:6827/1536802493] up:rejoin 2021-11-20T05:35:02.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:02 smithi058 conmon[33537]: cluster 2021-11-20T05:35:01.297669+0000 mon.smithi058 (mon.0) 569 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:02.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:02 smithi187 conmon[36358]: cluster 2021-11-20T05:35:01.297597+0000 mon.smithi058 (mon.0) 568 : 2021-11-20T05:35:02.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:02 smithi187 conmon[36358]: cluster [DBG] mds.? [v2:172.21.15.58:6826/1536802493,v1:172.21.15.58:6827/1536802493] up:rejoin 2021-11-20T05:35:02.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:02 smithi187 conmon[36358]: cluster 2021-11-20T05:35:01.297669+0000 mon.smithi058 (mon.0) 569 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:03.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:03 smithi187 conmon[36358]: cluster 2021-11-20T05:35:02.227064+0000 mgr.smithi058.zagdlk (mgr.14164) 220 : cluster 2021-11-20T05:35:03.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:03 smithi187 conmon[36358]: [DBG] pgmap v184: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T05:35:03.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:03 smithi187 conmon[36358]: cluster 2021-11-20T05:35:02.302296+0000 mon.smithi058 (mon.0) 570 : cluster [DBG] mds.? [v2:172.21.15.187:6824/4174217961,v1:172.21.15.187:6825/4174217961] up:standby 2021-11-20T05:35:03.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:03 smithi187 conmon[36358]: cluster 2021-11-20T05:35:02.302361+0000 mon.smithi058 (mon.0) 571 : cluster 2021-11-20T05:35:03.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:03 smithi187 conmon[36358]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:03.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:03 smithi058 conmon[33537]: cluster 2021-11-20T05:35:02.227064+0000 mgr.smithi058.zagdlk (mgr.14164) 220 : cluster 2021-11-20T05:35:03.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:03 smithi058 conmon[33537]: [DBG] pgmap v184: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T05:35:03.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:03 smithi058 conmon[33537]: cluster 2021-11-20T05:35:02.302296+0000 mon.smithi058 (mon.0) 570 : 2021-11-20T05:35:03.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:03 smithi058 conmon[33537]: cluster [DBG] mds.? [v2:172.21.15.187:6824/4174217961,v1:172.21.15.187:6825/4174217961] up:standby 2021-11-20T05:35:03.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:03 smithi058 conmon[33537]: cluster 2021-11-20T05:35:03.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:03 smithi058 conmon[33537]: 2021-11-20T05:35:02.302361+0000 mon.smithi058 (mon.0) 571 2021-11-20T05:35:03.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:03 smithi058 conmon[33537]: : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:05.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:05 smithi187 conmon[36358]: cluster 2021-11-20T05:35:04.227742+0000 mgr.smithi058.zagdlk (mgr.14164) 221 : cluster 2021-11-20T05:35:05.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:05 smithi187 conmon[36358]: [DBG] pgmap v185: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:05.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:05 smithi058 conmon[33537]: cluster 2021-11-20T05:35:04.227742+0000 2021-11-20T05:35:05.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:05 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 221 : cluster [DBG] pgmap v185: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:07.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:07 smithi187 conmon[36358]: cluster 2021-11-20T05:35:06.228357+0000 mgr.smithi058.zagdlk (mgr.14164) 222 : cluster [DBG] 2021-11-20T05:35:07.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:07 smithi187 conmon[36358]: pgmap v186: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:07.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:07 smithi058 conmon[33537]: cluster 2021-11-20T05:35:06.228357+0000 mgr.smithi058.zagdlk (mgr.14164) 222 : cluster [DBG] pgmap v186: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:09.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:09 smithi187 conmon[36358]: cluster 2021-11-20T05:35:08.229001+0000 mgr.smithi058.zagdlk (mgr.14164) 223 : cluster 2021-11-20T05:35:09.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:09 smithi187 conmon[36358]: [DBG] pgmap v187: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:09.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:09 smithi058 conmon[33537]: cluster 2021-11-20T05:35:08.229001+0000 mgr.smithi058.zagdlk (mgr.14164) 223 : cluster [DBG] pgmap v187: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:11.591 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:35:11.592+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.58:6826/3420124164 2021-11-20T05:35:11.591 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:35:11.592+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T05:35:11.592 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:35:11.592+0000 7f9ab9ffb700 1 client.14552 ms_handle_refused on v2:172.21.15.58:6826/3420124164 2021-11-20T05:35:11.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:11 smithi187 conmon[36358]: cluster 2021-11-20T05:35:10.229451+0000 mgr.smithi058.zagdlk (mgr.14164) 224 : cluster [DBG] pgmap v188: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:35:11.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:11 smithi058 conmon[33537]: cluster 2021-11-20T05:35:10.229451+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:35:11.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:11 smithi058 conmon[33537]: 14164) 224 : cluster [DBG] pgmap v188: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:35:13.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:13 smithi187 conmon[36358]: cluster 2021-11-20T05:35:12.229917+0000 mgr.smithi058.zagdlk (mgr.14164) 225 : cluster [DBG] pgmap v189: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 0 op/s 2021-11-20T05:35:13.579 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:13 smithi187 conmon[36358]: cluster 2021-11-20T05:35:12.524193+0000 mon.smithi058 (mon.0) 572 : cluster [INF] daemon mds.cephfs.smithi058.rxapzd restarted 2021-11-20T05:35:13.579 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:13 smithi187 conmon[36358]: cluster 2021-11- 2021-11-20T05:35:13.580 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:13 smithi187 conmon[36358]: 20T05:35:12.528135+0000 mon.smithi058 (mon.0) 573 : cluster 2021-11-20T05:35:13.580 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:13 smithi187 conmon[36358]: [DBG] osdmap e45: 6 total, 6 up, 6 in 2021-11-20T05:35:13.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:13 smithi058 conmon[33537]: cluster 2021-11-20T05:35:12.229917+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:35:13.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:13 smithi058 conmon[33537]: ) 225 : cluster [DBG] pgmap v189: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 0 op/s 2021-11-20T05:35:13.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:13 smithi058 conmon[33537]: cluster 2021-11-20T05:35:12.524193+0000 mon.smithi058 (mon.0) 572 : cluster [INF] daemon mds.cephfs.smithi058.rxapzd restarted 2021-11-20T05:35:13.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:13 smithi058 conmon[33537]: cluster 2021-11-20T05:35:12.528135+0000 mon.smithi058 (mon.0 2021-11-20T05:35:13.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:13 smithi058 conmon[33537]: ) 573 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2021-11-20T05:35:14.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:14 smithi187 conmon[36358]: cluster 2021-11-20T05:35:13.319654+0000 mon.smithi058 (mon.0) 574 : cluster [DBG] 2021-11-20T05:35:14.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:14 smithi187 conmon[36358]: mds.? [v2:172.21.15.58:6826/2824396697,v1:172.21.15.58:6827/2824396697] up:boot 2021-11-20T05:35:14.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:14 smithi187 conmon[36358]: cluster 2021-11-20T05:35:13.319730+0000 mon.smithi058 (mon.0) 575 : cluster [INF] Standby daemon mds.cephfs.smithi187.fikisa assigned to filesystem cephfs as rank 0 2021-11-20T05:35:14.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:14 smithi187 conmon[36358]: cluster 2021-11-20T05:35:13. 2021-11-20T05:35:14.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:14 smithi187 conmon[36358]: 320149+0000 mon.smithi058 (mon.0) 576 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 3 up:standby, 1 failed 2021-11-20T05:35:14.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:14 smithi187 conmon[36358]: audit 2021 2021-11-20T05:35:14.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:14 smithi187 conmon[36358]: -11-20T05:35:13.320823+0000 2021-11-20T05:35:14.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:14 smithi187 conmon[36358]: mon.smithi058 (mon.0) 577 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi058.rxapzd"}]: dispatch 2021-11-20T05:35:14.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:14 smithi187 conmon[36358]: cluster 2021-11-20T05:35:14.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:14 smithi187 conmon[36358]: 2021-11-20T05: 2021-11-20T05:35:14.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:14 smithi187 conmon[36358]: 35:13.324632+0000 mon.smithi058 ( 2021-11-20T05:35:14.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:14 smithi187 conmon[36358]: mon.0) 578 : cluster [DBG] 2021-11-20T05:35:14.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:14 smithi187 conmon[36358]: fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:14.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:14 smithi058 conmon[33537]: cluster 2021-11-20T05:35:13.319654+0000 mon.smithi058 (mon.0) 574 2021-11-20T05:35:14.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:14 smithi058 conmon[33537]: : cluster [DBG] mds.? [v2:172.21.15.58:6826/2824396697,v1:172.21.15.58:6827/2824396697] up:boot 2021-11-20T05:35:14.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:14 smithi058 conmon[33537]: cluster 2021-11-20T05:35:13.319730+0000 mon.smithi058 (mon.0) 575 : cluster [INF] Standby daemon mds.cephfs.smithi187.fikisa assigned to filesystem cephfs as rank 0 2021-11-20T05:35:14.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:14 smithi058 conmon[33537]: cluster 2021-11-20T05:35:13.320149+0000 mon.smithi058 2021-11-20T05:35:14.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:14 smithi058 conmon[33537]: (mon.0) 576 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 3 up:standby, 1 failed 2021-11-20T05:35:14.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:14 smithi058 conmon[33537]: audit 2021-11-20T05:35:14.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:14 smithi058 conmon[33537]: 2021-11-20T05:35: 2021-11-20T05:35:14.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:14 smithi058 conmon[33537]: 13.320823+0000 mon.smithi058 (mon.0) 577 : audit 2021-11-20T05:35:14.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:14 smithi058 conmon[33537]: [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi058.rxapzd"}]: dispatch 2021-11-20T05:35:14.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:14 smithi058 conmon[33537]: cluster 2021-11-20T05:35 2021-11-20T05:35:14.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:14 smithi058 conmon[33537]: :13.324632+0000 mon.smithi058 (mon.0) 578 : cluster 2021-11-20T05:35:14.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:14 smithi058 conmon[33537]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:15.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:15 smithi187 conmon[36358]: cluster 2021-11-20T05:35:14.230382+0000 mgr.smithi058.zagdlk (mgr.14164) 226 : cluster [DBG] pgmap v191: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T05:35:15.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:15 smithi187 conmon[36358]: cluster 2021-11-20T05:35:14. 2021-11-20T05:35:15.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:15 smithi187 conmon[36358]: 328289+0000 mon.smithi058 (mon.0) 579 : cluster [DBG] mds.? [v2:172.21.15.187:6826/2997529975,v1:172.21.15.187:6827/2997529975] up:resolve 2021-11-20T05:35:15.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:15 smithi187 conmon[36358]: cluster 2021-11-20T05:35:14.328386+0000 mon.smithi058 (mon.0) 580 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:15.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:15 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:35:15.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:15 smithi058 conmon[33537]: -11-20T05:35:14.230382+0000 mgr.smithi058.zagdlk (mgr.14164) 226 : cluster [DBG] pgmap v191: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T05:35:15.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:15 smithi058 conmon[33537]: cluster 2021-11-20T05:35:14.328289 2021-11-20T05:35:15.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:15 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 579 : cluster [DBG] mds.? [v2:172.21.15.187:6826/2997529975,v1:172.21.15.187:6827/2997529975] up:resolve 2021-11-20T05:35:15.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:15 smithi058 conmon[33537]: 2021-11-20T05:35:15.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:15 smithi058 conmon[33537]: cluster 2021-11-20T05:35:15.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:15 smithi058 conmon[33537]: 2021-11-20T05:35:14.328386+0000 2021-11-20T05:35:15.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:15 smithi058 conmon[33537]: mon.smithi058 (mon.0) 580 : cluster [DBG] 2021-11-20T05:35:15.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:15 smithi058 conmon[33537]: fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:16.666 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:16 smithi187 conmon[36358]: cluster 2021-11-20T05:35:15.329760+0000 mon.smithi058 (mon.0) 581 : cluster [DBG] mds.? [v2:172.21.15.187:6826/2997529975,v1:172.21.15.187:6827/2997529975] up:reconnect 2021-11-20T05:35:16.666 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:16 smithi187 conmon[36358]: cluster 2021-11-20T05:35 2021-11-20T05:35:16.667 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:16 smithi187 conmon[36358]: :15.329840+0000 mon.smithi058 (mon.0) 582 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:16.667 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:16 smithi187 conmon[36358]: cluster 2021-11-20T05:35:15.350086+0000 mds.cephfs.smithi187.fikisa (mds.0) 1 : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0219997 2021-11-20T05:35:16.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:16 smithi058 conmon[33537]: cluster 2021-11-20T05:35:15.329760+0000 mon.smithi058 (mon.0) 2021-11-20T05:35:16.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:16 smithi058 conmon[33537]: 581 : cluster [DBG] mds.? [v2:172.21.15.187:6826/2997529975,v1:172.21.15.187:6827/2997529975] up:reconnect 2021-11-20T05:35:16.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:16 smithi058 conmon[33537]: cluster 2021-11-20T05:35:15.329840+0000 mon.smithi058 (mon.0) 582 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:16.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:16 smithi058 conmon[33537]: cluster 2021-11-20T05:35:15.350086+0000 mds.cephfs.smithi187.fikisa (mds.0) 1 : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0219997 2021-11-20T05:35:17.523 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:17 smithi058 conmon[33537]: cluster 2021-11-20T05:35:16.231189+0000 mgr.smithi058.zagdlk (mgr.14164) 227 : cluster [DBG] pgmap v192: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T05:35:17.524 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:17 smithi058 conmon[33537]: cluster 2021-11-20T05:35:17.524 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:17 smithi058 conmon[33537]: 2021-11-20T05:35:16.335352+0000 mon.smithi058 (mon.0) 583 : cluster [DBG] mds.? [v2:172.21.15.187:6826/2997529975,v1:172.21.15.187:6827/2997529975] up:rejoin 2021-11-20T05:35:17.524 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:17 smithi058 conmon[33537]: cluster 2021-11-20T05:35:16.335451+0000 mon.smithi058 (mon.0) 584 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:17.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:17 smithi187 conmon[36358]: cluster 2021-11-20T05:35:16.231189+0000 mgr.smithi058.zagdlk (mgr.14164) 227 : cluster [DBG] pgmap v192: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T05:35:17.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:17 smithi187 conmon[36358]: cluster 2021-11-20T05:35:16.335352+0000 mon.smithi058 (mon.0) 583 : cluster [DBG] mds.? [v2:172.21.15.187:6826/2997529975,v1:172.21.15.187:6827/2997529975] up:rejoin 2021-11-20T05:35:17.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:17 smithi187 conmon[36358]: cluster 2021-11-20T05:35:16.335451+0000 mon.smithi058 (mon.0) 584 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:18.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:18 smithi187 conmon[36358]: cluster 2021-11-20T05:35:17.340432+0000 mon.smithi058 (mon.0) 585 : cluster [DBG] mds.? [v2:172.21.15.58:6826/2824396697,v1:172.21.15.58:6827/2824396697] up:standby 2021-11-20T05:35:18.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:18 smithi187 conmon[36358]: cluster 2021-11-20T05:35:17.340510+0000 mon.smithi058 (mon.0) 586 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:18.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:18 smithi058 conmon[33537]: cluster 2021-11-20T05:35:17.340432+0000 mon.smithi058 (mon.0) 585 2021-11-20T05:35:18.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:18 smithi058 conmon[33537]: : cluster [DBG] mds.? [v2:172.21.15.58:6826/2824396697,v1:172.21.15.58:6827/2824396697] up:standby 2021-11-20T05:35:18.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:18 smithi058 conmon[33537]: cluster 2021-11-20T05:35:17.340510+0000 mon.smithi058 (mon.0) 586 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:19.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:19 smithi187 conmon[36358]: cluster 2021-11-20T05:35:18.231830+0000 mgr.smithi058.zagdlk (mgr.14164) 228 : cluster [DBG] pgmap v193: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:19.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:19 smithi187 conmon[36358]: 2021-11-20T05:35:19.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:19 smithi058 conmon[33537]: cluster 2021-11-20T05:35:18.231830+0000 mgr.smithi058.zagdlk (mgr.14164) 228 : cluster [DBG] pgmap v193: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:21.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:21 smithi058 conmon[33537]: cluster 2021-11-20T05:35:20.232334+0000 mgr.smithi058.zagdlk (mgr.14164) 229 : 2021-11-20T05:35:21.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:21 smithi058 conmon[33537]: cluster [DBG] pgmap v194: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:21.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:21 smithi187 conmon[36358]: cluster 2021-11-20T05:35:20.232334+0000 mgr.smithi058.zagdlk (mgr.14164) 229 : cluster [DBG] pgmap v194: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:23.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:23 smithi187 conmon[36358]: cluster 2021-11-20T05:35:22.232923+0000 mgr.smithi058.zagdlk (mgr.14164) 230 : cluster [DBG] pgmap v195: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:23.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:23 smithi058 conmon[33537]: cluster 2021-11-20T05:35:22.232923+0000 mgr.smithi058.zagdlk (mgr.14164) 230 : cluster [DBG] pgmap v195: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:25.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:25 smithi187 conmon[36358]: cluster 2021-11-20T05:35:24.233589+0000 mgr.smithi058.zagdlk (mgr.14164) 231 : cluster [DBG] 2021-11-20T05:35:25.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:25 smithi187 conmon[36358]: pgmap v196: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:35:25.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:25 smithi058 conmon[33537]: cluster 2021-11-20T05:35:24.233589+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:35:25.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:25 smithi058 conmon[33537]: ) 231 : cluster [DBG] pgmap v196: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:35:26.606 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:35:26.607+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:35:26.607 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:35:26.608+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T05:35:26.607 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:35:26.608+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T05:35:26.608 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:35:26.608+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T05:35:26.608 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:35:26.608+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T05:35:27.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:27 smithi187 conmon[36358]: cluster 2021-11-20T05:35:26.234102+0000 mgr.smithi058.zagdlk (mgr.14164) 232 : cluster [DBG] pgmap v197: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 0 op/s 2021-11-20T05:35:27.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:27 smithi058 conmon[33537]: cluster 2021-11-20T05:35:26.234102+0000 mgr.smithi058.zagdlk (mgr.14164) 232 : cluster [DBG] pgmap v197: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 0 op/s 2021-11-20T05:35:29.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:29 smithi187 conmon[36358]: cluster 2021-11-20T05:35:28.234800+0000 mgr.smithi058.zagdlk (mgr.14164) 233 : cluster [DBG] pgmap v198: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 0 op/s 2021-11-20T05:35:29.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:29 smithi187 conmon[36358]: audit 2021-11-20T05:35:28.532630+0000 mon.smithi058 (mon.0) 587 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:35:29.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:29 smithi187 conmon[36358]: audit 2021-11-20T05:35:28.919406+0000 mon.smithi058 (mon.0) 588 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:35:29.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:29 smithi058 conmon[33537]: cluster 2021-11-20T05:35:28.234800+0000 mgr.smithi058.zagdlk (mgr.14164) 233 : cluster [DBG] pgmap v198: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 0 op/s 2021-11-20T05:35:29.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:29 smithi058 conmon[33537]: audit 2021-11-20T05:35:28.532630+0000 mon.smithi058 (mon.0) 587 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:35:29.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:29 smithi058 conmon[33537]: audit 2021-11-20T05:35:28.919406+0000 mon.smithi058 (mon.0) 588 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:35:31.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:31 smithi187 conmon[36358]: cluster 2021-11-20T05:35:30.235310+0000 mgr.smithi058.zagdlk (mgr.14164) 234 : cluster [DBG] pgmap v199: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:35:31.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:31 smithi187 conmon[36358]: cluster 2021-11-20T05:35:31.231408+0000 mon.smithi058 (mon.0) 589 : cluster [INF] daemon mds.cephfs.smithi187.fikisa restarted 2021-11-20T05:35:31.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:31 smithi187 conmon[36358]: cluster 2021-11-20T05:35:31.236503+0000 mon.smithi058 (mon.0) 590 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2021-11-20T05:35:31.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:31 smithi058 conmon[33537]: cluster 2021-11-20T05:35:30.235310+0000 mgr.smithi058.zagdlk (mgr.14164) 234 : cluster [DBG] pgmap v199: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:35:31.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:31 smithi058 conmon[33537]: cluster 2021-11- 2021-11-20T05:35:31.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:31 smithi058 conmon[33537]: 20T05:35:31.231408+0000 mon.smithi058 (mon.0) 589 : cluster [INF] daemon mds.cephfs.smithi187.fikisa restarted 2021-11-20T05:35:31.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:31 smithi058 conmon[33537]: cluster 2021-11-20T05:35:31.236503+0000 mon.smithi058 (mon.0) 590 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2021-11-20T05:35:32.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:32 smithi187 conmon[36358]: cluster 2021-11-20T05:35:31.371162+0000 mon.smithi058 (mon.0) 591 : cluster [DBG] mds.? [v2:172.21.15.187:6826/4049116176,v1:172.21.15.187:6827/4049116176] up:boot 2021-11-20T05:35:32.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:32 smithi187 conmon[36358]: cluster 2021-11-20T05:35:31.371224+0000 mon.smithi058 (mon.0) 592 : cluster [INF] Standby daemon mds.cephfs.smithi058.rxapzd assigned to filesystem cephfs as rank 0 2021-11-20T05:35:32.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:32 smithi187 conmon[36358]: cluster 2021-11-20T05:35:31.371636+0000 mon.smithi058 (mon.0) 593 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 3 up:standby, 1 failed 2021-11-20T05:35:32.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:32 smithi187 conmon[36358]: audit 2021-11-20T05:35:31.371777+0000 mon.smithi058 (mon.0) 594 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.fikisa"}]: dispatch 2021-11-20T05:35:32.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:32 smithi187 conmon[36358]: cluster 2021-11-20T05:35:31.375996+0000 mon.smithi058 (mon.0) 595 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:32.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:32 smithi058 conmon[33537]: cluster 2021-11-20T05:35:31.371162+0000 2021-11-20T05:35:32.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:32 smithi058 conmon[33537]: mon.smithi058 (mon.0) 591 : cluster [DBG] mds.? [v2:172.21.15.187:6826/4049116176,v1:172.21.15.187:6827/4049116176] up:boot 2021-11-20T05:35:32.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:32 smithi058 conmon[33537]: cluster 2021-11-20T05:35:31.371224+0000 mon.smithi058 ( 2021-11-20T05:35:32.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:32 smithi058 conmon[33537]: mon.0) 592 : cluster [INF] Standby daemon mds.cephfs.smithi058.rxapzd assigned to filesystem cephfs as rank 0 2021-11-20T05:35:32.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:32 smithi058 conmon[33537]: cluster 2021-11 2021-11-20T05:35:32.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:32 smithi058 conmon[33537]: -20T05:35:31.371636+0000 mon.smithi058 (mon.0) 593 2021-11-20T05:35:32.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:32 smithi058 conmon[33537]: : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 3 up:standby, 1 failed 2021-11-20T05:35:32.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:32 smithi058 conmon[33537]: audit 2021 2021-11-20T05:35:32.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:32 smithi058 conmon[33537]: -11-20T05:35:31.371777+0000 mon.smithi058 (mon.0) 594 : audit 2021-11-20T05:35:32.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:32 smithi058 conmon[33537]: [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.fikisa"}]: dispatch 2021-11-20T05:35:32.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:32 smithi058 conmon[33537]: cluster 2021-11-20T 2021-11-20T05:35:32.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:32 smithi058 conmon[33537]: 05:35:31.375996+0000 mon.smithi058 (mon.0) 2021-11-20T05:35:32.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:32 smithi058 conmon[33537]: 595 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:33.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:33 smithi187 conmon[36358]: cluster 2021-11-20T05:35:32.236023+0000 mgr.smithi058.zagdlk (mgr.14164) 235 2021-11-20T05:35:33.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:33 smithi187 conmon[36358]: : cluster [DBG] pgmap v201: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T05:35:33.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:33 smithi187 conmon[36358]: cluster 2021-11-20T05:35:32.381717+0000 mon.smithi058 (mon.0) 596 : cluster [DBG] mds.? [v2:172.21.15.58:6826/2824396697,v1:172.21.15.58:6827/2824396697] up:resolve 2021-11-20T05:35:33.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:33 smithi187 conmon[36358]: cluster 2021-11-20T05: 2021-11-20T05:35:33.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:33 smithi187 conmon[36358]: 35:32.381795+0000 mon.smithi058 (mon.0) 597 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:33.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:33 smithi058 conmon[33537]: cluster 2021-11-20T05: 2021-11-20T05:35:33.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:33 smithi058 conmon[33537]: 35:32.236023+0000 mgr.smithi058.zagdlk (mgr.14164) 235 : cluster [DBG] pgmap v201: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T05:35:33.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:33 smithi058 conmon[33537]: cluster 2021-11-20T05:35:33.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:33 smithi058 conmon[33537]: 2021-11-20T05:35 2021-11-20T05:35:33.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:33 smithi058 conmon[33537]: :32.381717+0000 mon.smithi058 (mon. 2021-11-20T05:35:33.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:33 smithi058 conmon[33537]: 0) 596 : cluster [DBG] mds.? [v2:172.21.15.58:6826/2824396697,v1:172.21.15.58:6827/2824396697] up:resolve 2021-11-20T05:35:33.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:33 smithi058 conmon[33537]: cluster 2021- 2021-11-20T05:35:33.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:33 smithi058 conmon[33537]: 11-20T05:35:32.381795 2021-11-20T05:35:33.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:33 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 597 : cluster 2021-11-20T05:35:33.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:33 smithi058 conmon[33537]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:34.450 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:34 smithi058 conmon[33537]: cluster 2021-11-20T05:35:33.386195+0000 mon.smithi058 (mon.0) 2021-11-20T05:35:34.450 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:34 smithi058 conmon[33537]: 598 : cluster [DBG] mds.? [v2:172.21.15.58:6826/2824396697,v1:172.21.15.58:6827/2824396697] up:reconnect 2021-11-20T05:35:34.450 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:34 smithi058 conmon[33537]: cluster 2021-11-20T05:35:33.386262+0000 mon.smithi058 (mon.0) 599 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:34.451 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:34 smithi058 conmon[33537]: cluster 2021-11-20T05:35:33.405831+0000 mds.cephfs.smithi058.rxapzd (mds.0) 1 : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0209997 2021-11-20T05:35:34.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:34 smithi187 conmon[36358]: cluster 2021-11-20T05:35:33.386195+0000 mon.smithi058 (mon.0) 598 : cluster [DBG] mds.? [v2:172.21.15.58:6826/2824396697,v1:172.21.15.58:6827/2824396697] up:reconnect 2021-11-20T05:35:34.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:34 smithi187 conmon[36358]: cluster 2021-11-20T05:35:33.386262+0000 mon.smithi058 (mon.0) 599 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:34.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:34 smithi187 conmon[36358]: cluster 2021-11-20T05:35:33.405831+0000 mds.cephfs.smithi058.rxapzd (mds.0) 1 : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0209997 2021-11-20T05:35:35.489 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:35 smithi058 conmon[33537]: cluster 2021-11-20T05:35:34.236486+0000 mgr.smithi058.zagdlk (mgr.14164) 236 : cluster 2021-11-20T05:35:35.489 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:35 smithi058 conmon[33537]: [DBG] pgmap v202: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:35.489 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:35 smithi058 conmon[33537]: cluster 2021-11-20T05:35:34.391229+0000 mon.smithi058 (mon.0) 600 : cluster [DBG] mds.? [v2:172.21.15.58:6826/2824396697,v1:172.21.15.58:6827/2824396697] up:rejoin 2021-11-20T05:35:35.489 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:35 smithi058 conmon[33537]: cluster 2021-11-20T05:35:34.391293+0000 mon.smithi058 (mon.0 2021-11-20T05:35:35.490 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:35 smithi058 conmon[33537]: ) 601 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:35.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:35 smithi187 conmon[36358]: cluster 2021-11-20T05:35:34.236486+0000 mgr.smithi058.zagdlk (mgr 2021-11-20T05:35:35.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:35 smithi187 conmon[36358]: .14164) 236 : cluster [DBG] pgmap v202: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:35.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:35 smithi187 conmon[36358]: cluster 2021-11-20T05:35:34.391229+0000 mon.smithi058 (mon.0) 600 : cluster [DBG] mds.? [v2:172.21.15.58:6826/2824396697,v1:172.21.15.58:6827/2824396697] up:rejoin 2021-11-20T05:35:35.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:35 smithi187 conmon[36358]: cluster 2021-11-20T05:35:35.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:35 smithi187 conmon[36358]: 2021-11-20T05:35:34.391293+0000 mon.smithi058 (mon.0) 601 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:36.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:36 smithi187 conmon[36358]: cluster 2021-11-20T05:35:35.392907+0000 mon.smithi058 (mon.0) 602 : cluster [DBG] mds.? [v2:172.21.15.187:6826/4049116176,v1:172.21.15.187:6827/4049116176] up:standby 2021-11-20T05:35:36.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:36 smithi187 conmon[36358]: cluster 2021-11-20T05:35:35.392969+0000 mon.smithi058 (mon.0) 603 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:36.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:36 smithi058 conmon[33537]: cluster 2021-11-20T05:35:35.392907+0000 mon.smithi058 (mon.0) 602 : cluster [DBG] mds.? [v2:172.21.15.187:6826/4049116176,v1:172.21.15.187:6827/4049116176] up:standby 2021-11-20T05:35:36.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:36 smithi058 conmon[33537]: cluster 2021-11-20 2021-11-20T05:35:36.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:36 smithi058 conmon[33537]: T05:35:35.392969+0000 mon.smithi058 (mon.0) 603 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:37.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:37 smithi058 conmon[33537]: cluster 2021-11-20T05:35:36.236960+0000 mgr.smithi058.zagdlk (mgr.14164) 237 : cluster 2021-11-20T05:35:37.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:37 smithi058 conmon[33537]: [DBG] pgmap v203: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:37.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:37 smithi187 conmon[36358]: cluster 2021-11-20T05:35:36.236960+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:35:37.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:37 smithi187 conmon[36358]: 14164) 237 : cluster [DBG] pgmap v203: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:39.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:39 smithi187 conmon[36358]: cluster 2021-11-20T05:35:38.237507+0000 mgr.smithi058.zagdlk (mgr.14164) 238 : cluster [DBG] pgmap v204: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:39.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:39 smithi058 conmon[33537]: cluster 2021-11-20T 2021-11-20T05:35:39.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:39 smithi058 conmon[33537]: 05:35:38.237507+0000 mgr.smithi058.zagdlk (mgr.14164) 238 : cluster [DBG] pgmap v204: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:41.617 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:35:41.619+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.58:6826/3420124164 2021-11-20T05:35:41.618 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:35:41.619+0000 7f9ab9ffb700 1 client.14552 ms_handle_refused on v2:172.21.15.58:6826/3420124164 2021-11-20T05:35:41.618 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:35:41.619+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T05:35:41.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:41 smithi187 conmon[36358]: cluster 2021-11-20T05:35:40.238092+0000 mgr.smithi058.zagdlk (mgr.14164) 239 : cluster [DBG] pgmap v205: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:41.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:41 smithi058 conmon[33537]: cluster 2021-11-20T05:35:40.238092+0000 mgr.smithi058.zagdlk (mgr.14164) 239 : cluster [DBG] pgmap v205: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:43.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:43 smithi187 conmon[36358]: cluster 2021-11-20T05:35:42.238832+0000 mgr.smithi058.zagdlk (mgr.14164) 240 : cluster 2021-11-20T05:35:43.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:43 smithi187 conmon[36358]: [DBG] pgmap v206: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T05:35:43.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:43 smithi058 conmon[33537]: cluster 2021-11-20T05:35:42.238832+0000 mgr.smithi058.zagdlk (mgr.14164) 240 : 2021-11-20T05:35:43.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:43 smithi058 conmon[33537]: cluster [DBG] pgmap v206: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T05:35:45.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:45 smithi058 conmon[33537]: cluster 2021-11-20T05:35:44.239314+0000 mgr.smithi058.zagdlk (mgr.14164) 241 : cluster 2021-11-20T05:35:45.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:45 smithi058 conmon[33537]: [DBG] pgmap v207: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 0 op/s 2021-11-20T05:35:45.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:45 smithi058 conmon[33537]: cluster 2021-11-20T05:35:45.219250+0000 mon.smithi058 (mon.0) 604 : cluster [INF] daemon mds.cephfs.smithi058.rxapzd restarted 2021-11-20T05:35:45.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:45 smithi058 conmon[33537]: cluster 2021-11-20T05:35:45 2021-11-20T05:35:45.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:45 smithi058 conmon[33537]: .222690+0000 mon.smithi058 (mon.0) 605 : cluster 2021-11-20T05:35:45.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:45 smithi058 conmon[33537]: [DBG] osdmap e47: 6 total, 6 up, 6 in 2021-11-20T05:35:45.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:45 smithi187 conmon[36358]: cluster 2021-11-20T05:35:44.239314+0000 mgr.smithi058.zagdlk (mgr.14164) 241 : cluster [DBG] 2021-11-20T05:35:45.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:45 smithi187 conmon[36358]: pgmap v207: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 0 op/s 2021-11-20T05:35:45.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:45 smithi187 conmon[36358]: cluster 2021-11-20T05:35:45.219250+0000 mon.smithi058 (mon.0) 604 : cluster [INF] daemon mds.cephfs.smithi058.rxapzd restarted 2021-11-20T05:35:45.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:45 smithi187 conmon[36358]: cluster 2021-11-20T05:35:45.222690+0000 mon.smithi058 (mon.0) 605 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2021-11-20T05:35:46.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:46 smithi187 conmon[36358]: cluster 2021-11-20T05:35:45.244448+0000 mon.smithi058 (mon.0) 606 : cluster 2021-11-20T05:35:46.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:46 smithi187 conmon[36358]: [DBG] mds.? [v2:172.21.15.58:6826/1753941223,v1:172.21.15.58:6827/1753941223] up:boot 2021-11-20T05:35:46.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:46 smithi187 conmon[36358]: cluster 2021-11-20T05:35:45.244499+0000 mon.smithi058 (mon.0) 607 : cluster [INF] Standby daemon mds.cephfs.smithi187.wxduhu assigned to filesystem cephfs as rank 0 2021-11-20T05:35:46.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:46 smithi187 conmon[36358]: cluster 2021-11-20T05:35:45.244733+0000 mon.smithi058 (mon.0) 2021-11-20T05:35:46.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:46 smithi187 conmon[36358]: 608 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 3 up:standby, 1 failed 2021-11-20T05:35:46.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:46 smithi187 conmon[36358]: audit 2021-11-20T 2021-11-20T05:35:46.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:46 smithi187 conmon[36358]: 05:35:45.244946+0000 mon.smithi058 (mon.0) 609 : 2021-11-20T05:35:46.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:46 smithi187 conmon[36358]: audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi058.rxapzd"}]: dispatch 2021-11-20T05:35:46.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:46 smithi187 conmon[36358]: cluster 2021-11-20T05 2021-11-20T05:35:46.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:46 smithi187 conmon[36358]: :35:45.247369+0000 mon.smithi058 (mon.0) 610 : cluster 2021-11-20T05:35:46.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:46 smithi187 conmon[36358]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:46.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:46 smithi058 conmon[33537]: cluster 2021-11-20T05:35:45.244448+0000 mon.smithi058 (mon 2021-11-20T05:35:46.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:46 smithi058 conmon[33537]: .0) 606 : cluster [DBG] mds.? [v2:172.21.15.58:6826/1753941223,v1:172.21.15.58:6827/1753941223] up:boot 2021-11-20T05:35:46.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:46 smithi058 conmon[33537]: cluster 2021-11-20T05:35:46.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:46 smithi058 conmon[33537]: 2021-11-20T05:35:45.244499+0000 mon.smithi058 (mon.0) 607 : cluster [INF] Standby daemon mds.cephfs.smithi187.wxduhu assigned to filesystem cephfs as rank 0 2021-11-20T05:35:46.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:46 smithi058 conmon[33537]: cluster 2021-11-20T05: 2021-11-20T05:35:46.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:46 smithi058 conmon[33537]: 35:45.244733+0000 mon.smithi058 (mon.0) 608 : 2021-11-20T05:35:46.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:46 smithi058 conmon[33537]: cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 3 up:standby, 1 failed 2021-11-20T05:35:46.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:46 smithi058 conmon[33537]: audit 2021-11-20T 2021-11-20T05:35:46.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:46 smithi058 conmon[33537]: 05:35:45.244946+0000 mon.smithi058 (mon 2021-11-20T05:35:46.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:46 smithi058 conmon[33537]: .0) 609 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi058.rxapzd"}]: dispatch 2021-11-20T05:35:46.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:46 smithi058 conmon[33537]: cluster 2021-11- 2021-11-20T05:35:46.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:46 smithi058 conmon[33537]: 20T05:35:45.247369+0000 mon.smithi058 2021-11-20T05:35:46.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:46 smithi058 conmon[33537]: (mon.0) 610 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:47.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:47 smithi187 conmon[36358]: cluster 2021-11-20T05:35:46.239903+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:35:47.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:47 smithi187 conmon[36358]: 242 : cluster [DBG] pgmap v209: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:35:47.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:47 smithi187 conmon[36358]: cluster 2021-11-20T05:35:46.253866+0000 mon.smithi058 (mon.0) 611 : cluster [DBG] mds.? [v2:172.21.15.187:6824/4174217961,v1:172.21.15.187:6825/4174217961] up:resolve 2021-11-20T05:35:47.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:47 smithi187 conmon[36358]: cluster 2021-11-20T05:35:46.253971 2021-11-20T05:35:47.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:47 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 612 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:47.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:47 smithi058 conmon[33537]: cluster 2021-11-20T05:35:46.239903+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:35:47.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:47 smithi058 conmon[33537]: 242 : cluster [DBG] pgmap v209: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:35:47.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:47 smithi058 conmon[33537]: cluster 2021-11-20T05:35:46.253866+0000 mon.smithi058 (mon.0) 611 : cluster [DBG] mds.? [v2:172.21.15.187:6824/4174217961,v1:172.21.15.187:6825/4174217961] up:resolve 2021-11-20T05:35:47.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:47 smithi058 conmon[33537]: cluster 2021-11 2021-11-20T05:35:47.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:47 smithi058 conmon[33537]: -20T05:35:46.253971+0000 mon.smithi058 (mon.0) 612 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:48.663 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:48 smithi187 conmon[36358]: cluster 2021-11-20T05:35:47.255735+0000 2021-11-20T05:35:48.664 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:48 smithi187 conmon[36358]: mon.smithi058 (mon.0) 613 : cluster [DBG] mds.? [v2:172.21.15.187:6824/4174217961,v1:172.21.15.187:6825/4174217961] up:reconnect 2021-11-20T05:35:48.664 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:48 smithi187 conmon[36358]: cluster 2021-11-20T05:35:47.255816+0000 2021-11-20T05:35:48.664 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:48 smithi187 conmon[36358]: mon.smithi058 (mon.0) 614 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:48.665 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:48 smithi187 conmon[36358]: cluster 2021-11 2021-11-20T05:35:48.665 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:48 smithi187 conmon[36358]: -20T05:35:47.275591+0000 2021-11-20T05:35:48.665 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:48 smithi187 conmon[36358]: mds.cephfs.smithi187.wxduhu (mds.0) 1 : cluster 2021-11-20T05:35:48.665 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:48 smithi187 conmon[36358]: [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0219997 2021-11-20T05:35:48.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:48 smithi058 conmon[33537]: cluster 2021-11-20T05:35:47.255735+0000 mon.smithi058 (mon.0) 613 2021-11-20T05:35:48.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:48 smithi058 conmon[33537]: : cluster [DBG] mds.? [v2:172.21.15.187:6824/4174217961,v1:172.21.15.187:6825/4174217961] up:reconnect 2021-11-20T05:35:48.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:48 smithi058 conmon[33537]: cluster 2021-11-20T05:35:47.255816+0000 mon.smithi058 (mon.0) 614 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:48.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:48 smithi058 conmon[33537]: cluster 2021-11-20T05:35:47.275591+0000 2021-11-20T05:35:48.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:48 smithi058 conmon[33537]: mds.cephfs.smithi187.wxduhu (mds.0) 1 : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0219997 2021-11-20T05:35:48.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:48 smithi058 conmon[33537]: 2021-11-20T05:35:49.528 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:49 smithi058 conmon[33537]: cluster 2021-11-20T05:35:48.243509+0000 mgr.smithi058.zagdlk (mgr.14164) 243 : cluster [DBG] pgmap v210: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2021-11-20T05:35:49.528 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:49 smithi058 conmon[33537]: cluster 2021-11-20T05:35:48.263702+0000 mon.smithi058 (mon.0) 615 : cluster [DBG] mds.? [v2:172.21.15.187:6824/4174217961,v1:172.21.15.187:6825/4174217961] up:rejoin 2021-11-20T05:35:49.528 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:49 smithi058 conmon[33537]: cluster 2021-11-20T05:35:48.263810+0000 mon.smithi058 (mon.0) 616 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:49.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:49 smithi187 conmon[36358]: cluster 2021-11-20T05:35:48.243509+0000 mgr.smithi058.zagdlk (mgr.14164) 243 : cluster [DBG] pgmap v210: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2021-11-20T05:35:49.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:49 smithi187 conmon[36358]: cluster 2021-11-20T05:35:48.263702 2021-11-20T05:35:49.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:49 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 615 : cluster [DBG] mds.? [v2:172.21.15.187:6824/4174217961,v1:172.21.15.187:6825/4174217961] up:rejoin 2021-11-20T05:35:49.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:49 smithi187 conmon[36358]: cluster 2021-11-20T05:35:48.263810+0000 mon.smithi058 (mon.0) 616 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:50.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:50 smithi058 conmon[33537]: cluster 2021-11-20T05:35:49.269605+0000 mon.smithi058 (mon.0) 2021-11-20T05:35:50.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:50 smithi058 conmon[33537]: 617 : cluster [DBG] mds.? [v2:172.21.15.58:6826/1753941223,v1:172.21.15.58:6827/1753941223] up:standby 2021-11-20T05:35:50.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:50 smithi058 conmon[33537]: cluster 2021-11-20T05:35:49.269664+0000 mon.smithi058 (mon.0) 618 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:50.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:50 smithi187 conmon[36358]: cluster 2021-11-20T05:35:49.269605+0000 mon.smithi058 (mon.0) 617 : cluster [DBG] mds.? [v2:172.21.15.58:6826/1753941223,v1:172.21.15.58:6827/1753941223] up:standby 2021-11-20T05:35:50.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:50 smithi187 conmon[36358]: cluster 2021-11-20T05 2021-11-20T05:35:50.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:50 smithi187 conmon[36358]: :35:49.269664+0000 mon.smithi058 (mon.0) 618 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:51.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:51 smithi187 conmon[36358]: cluster 2021-11-20T05:35:50.244326+0000 mgr.smithi058.zagdlk (mgr.14164) 244 : cluster [DBG] pgmap v211: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:51.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:51 smithi187 conmon[36358]: 2021-11-20T05:35:51.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:51 smithi058 conmon[33537]: cluster 2021-11-20T05:35:50.244326+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:35:51.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:51 smithi058 conmon[33537]: ) 244 : cluster [DBG] pgmap v211: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:53.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:53 smithi058 conmon[33537]: cluster 2021-11-20T05:35:52.244940+0000 mgr.smithi058.zagdlk (mgr.14164) 245 : 2021-11-20T05:35:53.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:53 smithi058 conmon[33537]: cluster [DBG] pgmap v212: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:53.576 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:53 smithi187 conmon[36358]: cluster 2021-11-20T05:35:52.244940+0000 mgr.smithi058.zagdlk (mgr.14164) 245 : cluster [DBG] pgmap v212: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:53.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:53 smithi187 conmon[36358]: 2021-11-20T05:35:55.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:55 smithi187 conmon[36358]: cluster 2021-11-20T05:35:54.245438+0000 mgr.smithi058.zagdlk (mgr.14164) 246 : cluster [DBG] pgmap v213: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:55.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:55 smithi187 conmon[36358]: 2021-11-20T05:35:55.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:55 smithi058 conmon[33537]: cluster 2021-11-20T05:35:54.245438+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:35:55.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:55 smithi058 conmon[33537]: ) 246 : cluster [DBG] pgmap v213: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:35:56.625 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:35:56.627+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:35:56.626 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:35:56.627+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T05:35:56.626 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:35:56.627+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T05:35:56.627 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:35:56.627+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T05:35:56.627 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:35:56.627+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T05:35:57.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:57 smithi187 conmon[36358]: cluster 2021-11-20T05:35:56.246060+0000 mgr.smithi058.zagdlk (mgr.14164) 247 : cluster [DBG] 2021-11-20T05:35:57.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:57 smithi187 conmon[36358]: pgmap v214: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T05:35:57.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:57 smithi058 conmon[33537]: cluster 2021-11-20T05:35:56.246060+0000 2021-11-20T05:35:57.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:57 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 247 : cluster [DBG] pgmap v214: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T05:35:59.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: cluster 2021-11-20T05:35:58.246678+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:35:59.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: 14164) 248 : cluster [DBG] pgmap v215: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:35:59.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: cluster 2021-11-20T05:35:59.226236+0000 mon.smithi058 (mon.0) 619 : cluster [INF] daemon mds.cephfs.smithi187.wxduhu restarted 2021-11-20T05:35:59.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: cluster 2021-11-20T05:35: 2021-11-20T05:35:59.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: 59.230476+0000 mon.smithi058 (mon.0 2021-11-20T05:35:59.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: ) 620 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2021-11-20T05:35:59.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: cluster 2021-11 2021-11-20T05:35:59.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: -20T05:35:59.241920+0000 2021-11-20T05:35:59.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: mon.smithi058 (mon.0) 621 : cluster 2021-11-20T05:35:59.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: [INF] Standby daemon mds.cephfs.smithi058.rxapzd assigned to filesystem cephfs as rank 0 2021-11-20T05:35:59.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: cluster 2021 2021-11-20T05:35:59.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: -11-20T05 2021-11-20T05:35:59.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: :35:59.245200+0000 mon.smithi058 2021-11-20T05:35:59.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: (mon.0) 622 : cluster 2021-11-20T05:35:59.699 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: [DBG] mds.? [v2:172.21.15.187:6824/126010257,v1:172.21.15.187:6825/126010257] up:boot 2021-11-20T05:35:59.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: cluster 2021- 2021-11-20T05:35:59.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: 11-20T05:35:59.245266+0000 mon.smithi058 2021-11-20T05:35:59.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: (mon.0) 623 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:59.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: audit 2021-11 2021-11-20T05:35:59.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: -20T05:35:59.245522+0000 mon.smithi058 ( 2021-11-20T05:35:59.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:35:59 smithi187 conmon[36358]: mon.0) 624 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.wxduhu"}]: dispatch 2021-11-20T05:35:59.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: cluster 2021-11-20T05:35:58.246678+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:35:59.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: ) 248 : cluster [DBG] pgmap v215: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:35:59.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: cluster 2021-11-20T05:35:59.226236+0000 mon.smithi058 (mon.0) 619 : cluster [INF] daemon mds.cephfs.smithi187.wxduhu restarted 2021-11-20T05:35:59.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: cluster 2021-11-20T05:35:59.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: 2021-11-20T05:35:59.230476+0000 mon.smithi058 2021-11-20T05:35:59.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: (mon.0) 620 : cluster [DBG] 2021-11-20T05:35:59.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: osdmap e48: 6 total, 6 up, 6 in 2021-11-20T05:35:59.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: cluster 2021- 2021-11-20T05:35:59.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: 11-20T05:35:59.241920 2021-11-20T05:35:59.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 621 : cluster 2021-11-20T05:35:59.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: [INF] Standby daemon mds.cephfs.smithi058.rxapzd assigned to filesystem cephfs as rank 0 2021-11-20T05:35:59.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: cluster 2021-11-20T05:35:59.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: 2021-11-20T05 2021-11-20T05:35:59.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: :35:59.245200+0000 mon.smithi058 2021-11-20T05:35:59.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: (mon.0) 622 : cluster [DBG] 2021-11-20T05:35:59.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: mds.? [v2:172.21.15.187:6824/126010257,v1:172.21.15.187:6825/126010257] up:boot 2021-11-20T05:35:59.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: cluster 2021-11-20T05:35:59.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: 2021-11-20T05:35 2021-11-20T05:35:59.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: :59.245266+0000 mon.smithi058 (mon. 2021-11-20T05:35:59.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: 0) 623 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:35:59.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: audit 2021-11- 2021-11-20T05:35:59.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: 20T05:35:59.245522+0000 2021-11-20T05:35:59.724 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: mon.smithi058 (mon.0) 624 : audit [DBG] 2021-11-20T05:35:59.724 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:35:59 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.wxduhu"}]: dispatch 2021-11-20T05:36:01.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:01 smithi187 conmon[36358]: cluster 2021-11-20T05:36:00.247110+0000 mgr.smithi058.zagdlk (mgr.14164) 249 : cluster [DBG] pgmap v217: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:36:01.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:01 smithi187 conmon[36358]: 2021-11-20T05:36:01.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:01 smithi187 conmon[36358]: cluster 2021-11-20T05:36:00.262170+0000 mon.smithi058 (mon.0) 625 : cluster [DBG] mds.? [v2:172.21.15.58:6826/1753941223,v1:172.21.15.58:6827/1753941223] up:resolve 2021-11-20T05:36:01.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:01 smithi187 conmon[36358]: cluster 2021-11-20T05:36:00.262250+0000 mon.smithi058 (mon.0) 2021-11-20T05:36:01.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:01 smithi187 conmon[36358]: 626 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:01.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:01 smithi187 conmon[36358]: audit 2021-11-20T05:36 2021-11-20T05:36:01.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:01 smithi187 conmon[36358]: :00.386667+0000 mon.smithi058 (mon.0) 627 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:36:01.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:01 smithi187 conmon[36358]: audit 2021 2021-11-20T05:36:01.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:01 smithi187 conmon[36358]: -11-20T05:36:00.386879+0000 2021-11-20T05:36:01.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:01 smithi187 conmon[36358]: mon.smithi058 (mon.0) 628 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:36:01.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:01 smithi187 conmon[36358]: 2021-11-20T05:36:01.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:01 smithi058 conmon[33537]: cluster 2021-11-20T05:36:00.247110+0000 mgr.smithi058.zagdlk (mgr.14164) 249 : cluster 2021-11-20T05:36:01.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:01 smithi058 conmon[33537]: [DBG] pgmap v217: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:36:01.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:01 smithi058 conmon[33537]: cluster 2021-11-20T05:36:00.262170+0000 mon.smithi058 (mon.0) 625 : cluster [DBG] mds.? [v2:172.21.15.58:6826/1753941223,v1:172.21.15.58:6827/1753941223] up:resolve 2021-11-20T05:36:01.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:01 smithi058 conmon[33537]: cluster 2021-11-20T05:36:00.262250 2021-11-20T05:36:01.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:01 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 626 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:01.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:01 smithi058 conmon[33537]: audit 2021-11-20T05:36:00.386667+0000 mon.smithi058 ( 2021-11-20T05:36:01.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:01 smithi058 conmon[33537]: mon.0) 627 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:36:01.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:01 smithi058 conmon[33537]: audit 2021-11-20T05:36:00.386879+0000 2021-11-20T05:36:01.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:01 smithi058 conmon[33537]: mon.smithi058 (mon.0) 628 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:36:02.607 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:02 smithi058 conmon[33537]: cluster 2021-11-20T05:36:01. 2021-11-20T05:36:02.608 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:02 smithi058 conmon[33537]: 267461+0000 mon.smithi058 (mon.0) 629 : cluster [DBG] mds.? [v2:172.21.15.58:6826/1753941223,v1:172.21.15.58:6827/1753941223] up:reconnect 2021-11-20T05:36:02.608 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:02 smithi058 conmon[33537]: cluster 2021-11-20T05:36:01.267534+0000 mon.smithi058 (mon 2021-11-20T05:36:02.608 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:02 smithi058 conmon[33537]: .0) 630 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:02.608 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:02 smithi058 conmon[33537]: cluster 2021-11- 2021-11-20T05:36:02.609 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:02 smithi058 conmon[33537]: 20T05:36:01.287029+0000 mds.cephfs.smithi058.rxapzd (mds.0) 1 : cluster 2021-11-20T05:36:02.609 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:02 smithi058 conmon[33537]: [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0229996 2021-11-20T05:36:02.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:02 smithi187 conmon[36358]: cluster 2021-11-20T05:36:01.267461+0000 mon.smithi058 (mon.0) 629 : cluster [DBG] mds.? [v2:172.21.15.58:6826/1753941223,v1:172.21.15.58:6827/1753941223] up:reconnect 2021-11-20T05:36:02.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:02 smithi187 conmon[36358]: cluster 2021-11-20T05:36:01.267534 2021-11-20T05:36:02.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:02 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 630 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:02.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:02 smithi187 conmon[36358]: cluster 2021-11-20T05:36:01.287029+0000 mds.cephfs.smithi058.rxapzd (mds.0) 1 : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0229996 2021-11-20T05:36:03.520 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:03 smithi058 conmon[33537]: cluster 2021-11-20T05:36:02.251176+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:36:03.521 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:03 smithi058 conmon[33537]: 250 : cluster [DBG] pgmap v218: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T05:36:03.521 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:03 smithi058 conmon[33537]: cluster 2021-11-20T05:36:02.272098+0000 mon.smithi058 (mon.0) 631 : cluster [DBG] mds.? [v2:172.21.15.58:6826/1753941223,v1:172.21.15.58:6827/1753941223] up:rejoin 2021-11-20T05:36:03.521 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:03 smithi058 conmon[33537]: cluster 2021-11-20T05:36:02. 2021-11-20T05:36:03.521 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:03 smithi058 conmon[33537]: 272175+0000 mon.smithi058 (mon.0) 632 : cluster 2021-11-20T05:36:03.522 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:03 smithi058 conmon[33537]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:03.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:03 smithi187 conmon[36358]: cluster 2021-11-20T05:36:02.251176+0000 mgr.smithi058.zagdlk (mgr.14164) 250 : cluster [DBG] pgmap v218: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T05:36:03.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:03 smithi187 conmon[36358]: cluster 2021-11-20T05:36:02.272098+0000 mon.smithi058 (mon.0) 631 : cluster [DBG] mds.? [v2:172.21.15.58:6826/1753941223,v1:172.21.15.58:6827/1753941223] up:rejoin 2021-11-20T05:36:03.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:03 smithi187 conmon[36358]: cluster 2021-11-20T05:36:02.272175+0000 mon.smithi058 (mon.0) 632 : 2021-11-20T05:36:03.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:03 smithi187 conmon[36358]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:04.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:04 smithi058 conmon[33537]: cluster 2021-11-20T05:36:03.281394+0000 2021-11-20T05:36:04.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:04 smithi058 conmon[33537]: mon.smithi058 (mon.0) 633 : cluster [DBG] mds.? [v2:172.21.15.187:6824/126010257,v1:172.21.15.187:6825/126010257] up:standby 2021-11-20T05:36:04.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:04 smithi058 conmon[33537]: cluster 2021-11-20T05:36:03.281470+0000 mon.smithi058 (mon.0) 634 : cluster 2021-11-20T05:36:04.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:04 smithi058 conmon[33537]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:04.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:04 smithi187 conmon[36358]: cluster 2021-11-20T05:36:03.281394+0000 mon.smithi058 (mon.0) 633 : cluster 2021-11-20T05:36:04.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:04 smithi187 conmon[36358]: [DBG] mds.? [v2:172.21.15.187:6824/126010257,v1:172.21.15.187:6825/126010257] up:standby 2021-11-20T05:36:04.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:04 smithi187 conmon[36358]: cluster 2021-11-20T05:36:03.281470+0000 mon.smithi058 (mon.0) 634 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:05.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:05 smithi187 conmon[36358]: cluster 2021-11-20T05:36:04.252593+0000 mgr.smithi058.zagdlk (mgr.14164) 251 : cluster [DBG] 2021-11-20T05:36:05.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:05 smithi187 conmon[36358]: pgmap v219: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:05.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:05 smithi058 conmon[33537]: cluster 2021-11-20T05:36:04.252593+0000 mgr.smithi058.zagdlk (mgr.14164) 251 : cluster [DBG] pgmap v219: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:07.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:07 smithi187 conmon[36358]: cluster 2021-11-20T05:36:06.253264+0000 mgr.smithi058.zagdlk (mgr.14164) 252 : cluster [DBG] 2021-11-20T05:36:07.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:07 smithi187 conmon[36358]: pgmap v220: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:07.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:07 smithi058 conmon[33537]: cluster 2021-11-20T05:36:06 2021-11-20T05:36:07.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:07 smithi058 conmon[33537]: .253264+0000 mgr.smithi058.zagdlk (mgr.14164) 252 : cluster [DBG] pgmap v220: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:09.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:09 smithi187 conmon[36358]: cluster 2021-11-20T05:36:08.253972+0000 mgr.smithi058.zagdlk (mgr.14164) 253 : cluster [DBG] pgmap v221: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:09.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:09 smithi187 conmon[36358]: 2021-11-20T05:36:09.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:09 smithi058 conmon[33537]: cluster 2021-11-20T05:36:08.253972+0000 mgr.smithi058.zagdlk (mgr.14164) 253 : cluster [DBG] pgmap v221: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:11.629 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:11.631+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.58:6826/3420124164 2021-11-20T05:36:11.629 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:11.631+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T05:36:11.629 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:11.631+0000 7f9ab9ffb700 1 client.14552 ms_handle_refused on v2:172.21.15.58:6826/3420124164 2021-11-20T05:36:11.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:11 smithi187 conmon[36358]: cluster 2021-11-20T05:36:10.254650+0000 mgr.smithi058.zagdlk (mgr.14164) 254 : cluster 2021-11-20T05:36:11.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:11 smithi187 conmon[36358]: [DBG] pgmap v222: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T05:36:11.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:11 smithi058 conmon[33537]: cluster 2021-11-20T05:36:10.254650+0000 mgr.smithi058.zagdlk (mgr.14164) 254 2021-11-20T05:36:11.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:11 smithi058 conmon[33537]: : cluster [DBG] pgmap v222: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T05:36:13.576 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:13 smithi187 conmon[36358]: cluster 2021-11-20T05:36:12.255189+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:36:13.576 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:13 smithi187 conmon[36358]: mgr.14164) 255 : cluster [DBG] pgmap v223: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:36:13.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:13 smithi187 conmon[36358]: cluster 2021-11-20T05:36:13.260917+0000 mon.smithi058 (mon.0) 635 : cluster [INF] daemon mds.cephfs.smithi058.rxapzd restarted 2021-11-20T05:36:13.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:13 smithi187 conmon[36358]: cluster 2021-11-20T05 2021-11-20T05:36:13.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:13 smithi187 conmon[36358]: :36:13.264585+0000 mon.smithi058 (mon.0) 636 : cluster [DBG] 2021-11-20T05:36:13.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:13 smithi187 conmon[36358]: osdmap e49: 6 total, 6 up, 6 in 2021-11-20T05:36:13.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:13 smithi058 conmon[33537]: cluster 2021-11-20T05:36:12.255189+0000 2021-11-20T05:36:13.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:13 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 255 : cluster [DBG] pgmap v223: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:36:13.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:13 smithi058 conmon[33537]: cluster 2021-11-20T05:36:13.260917+0000 mon.smithi058 (mon.0) 635 : cluster [INF] daemon mds.cephfs.smithi058.rxapzd restarted 2021-11-20T05:36:13.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:13 smithi058 conmon[33537]: cluster 2021-11-20T05:36:13.264585+0000 mon.smithi058 (mon 2021-11-20T05:36:13.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:13 smithi058 conmon[33537]: .0) 636 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2021-11-20T05:36:14.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:14 smithi187 conmon[36358]: cluster 2021-11-20T05:36:13.297426+0000 mon.smithi058 (mon.0) 637 : cluster [DBG] mds.? [v2:172.21.15.58:6826/444955435,v1:172.21.15.58:6827/444955435] up:boot 2021-11-20T05:36:14.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:14 smithi187 conmon[36358]: 2021-11-20T05:36:14.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:14 smithi187 conmon[36358]: cluster 2021-11-20T05:36:13.297497+0000 mon.smithi058 (mon.0) 638 : cluster [INF] Standby daemon mds.cephfs.smithi187.fikisa assigned to filesystem cephfs as rank 0 2021-11-20T05:36:14.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:14 smithi187 conmon[36358]: cluster 2021-11-20T05:36 2021-11-20T05:36:14.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:14 smithi187 conmon[36358]: :13.297797+0000 mon.smithi058 (mon.0) 639 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 3 up:standby, 1 failed 2021-11-20T05:36:14.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:14 smithi187 conmon[36358]: audit 2021-11-20 2021-11-20T05:36:14.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:14 smithi187 conmon[36358]: T05:36:13.297898+0000 mon.smithi058 (mon.0) 640 : audit [DBG] 2021-11-20T05:36:14.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:14 smithi187 conmon[36358]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi058.rxapzd"}]: dispatch 2021-11-20T05:36:14.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:14 smithi187 conmon[36358]: cluster 2021- 2021-11-20T05:36:14.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:14 smithi187 conmon[36358]: 11-20T05:36:13.301143+0000 mon.smithi058 (mon.0) 641 : 2021-11-20T05:36:14.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:14 smithi187 conmon[36358]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:14.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:14 smithi058 conmon[33537]: cluster 2021-11-20T05:36:13.297426+0000 mon.smithi058 (mon.0) 637 : cluster 2021-11-20T05:36:14.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:14 smithi058 conmon[33537]: [DBG] mds.? [v2:172.21.15.58:6826/444955435,v1:172.21.15.58:6827/444955435] up:boot 2021-11-20T05:36:14.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:14 smithi058 conmon[33537]: cluster 2021-11-20T05:36:13.297497+0000 mon.smithi058 (mon.0) 638 : cluster [INF] Standby daemon mds.cephfs.smithi187.fikisa assigned to filesystem cephfs as rank 0 2021-11-20T05:36:14.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:14 smithi058 conmon[33537]: cluster 2021-11-20T05 2021-11-20T05:36:14.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:14 smithi058 conmon[33537]: :36:13.297797+0000 mon.smithi058 2021-11-20T05:36:14.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:14 smithi058 conmon[33537]: (mon.0) 639 : cluster [DBG] 2021-11-20T05:36:14.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:14 smithi058 conmon[33537]: fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 3 up:standby, 1 failed 2021-11-20T05:36:14.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:14 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:36:14.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:14 smithi058 conmon[33537]: 11-20T05:36:13.297898+0000 mon.smithi058 2021-11-20T05:36:14.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:14 smithi058 conmon[33537]: (mon.0) 640 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi058.rxapzd"}]: dispatch 2021-11-20T05:36:14.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:14 smithi058 conmon[33537]: cluster 2021-11-20T05:36:14.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:14 smithi058 conmon[33537]: 2021-11-20T05:36 2021-11-20T05:36:14.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:14 smithi058 conmon[33537]: :13.301143+0000 mon.smithi058 (mon.0 2021-11-20T05:36:14.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:14 smithi058 conmon[33537]: ) 641 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:14.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:14 smithi058 conmon[33537]: 2021-11-20T05:36:15.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:15 smithi187 conmon[36358]: cluster 2021-11-20T05:36:14.255657+0000 mgr.smithi058.zagdlk (mgr.14164) 256 : cluster [DBG] pgmap v225: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T05:36:15.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:15 smithi187 conmon[36358]: cluster 2021-11-20T05:36:14. 2021-11-20T05:36:15.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:15 smithi187 conmon[36358]: 306465+0000 mon.smithi058 (mon.0) 642 : cluster [DBG] mds.? [v2:172.21.15.187:6826/4049116176,v1:172.21.15.187:6827/4049116176] up:resolve 2021-11-20T05:36:15.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:15 smithi187 conmon[36358]: cluster 2021-11-20T05:36:14.306534+0000 mon.smithi058 (mon.0) 643 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:15.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:15 smithi058 conmon[33537]: cluster 2021-11-20T05:36:14.255657+0000 mgr.smithi058.zagdlk (mgr.14164) 256 : cluster [DBG] pgmap v225: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T05:36:15.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:15 smithi058 conmon[33537]: 2021-11-20T05:36:15.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:15 smithi058 conmon[33537]: cluster 2021-11-20T05:36:14.306465+0000 mon.smithi058 (mon.0) 642 : cluster [DBG] mds.? [v2:172.21.15.187:6826/4049116176,v1:172.21.15.187:6827/4049116176] up:resolve 2021-11-20T05:36:15.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:15 smithi058 conmon[33537]: cluster 2021-11-20T 2021-11-20T05:36:15.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:15 smithi058 conmon[33537]: 05:36:14.306534+0000 mon.smithi058 (mon.0) 643 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:16.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:16 smithi187 conmon[36358]: cluster 2021-11-20T05:36:15.308168+0000 mon.smithi058 (mon.0 2021-11-20T05:36:16.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:16 smithi187 conmon[36358]: ) 644 : cluster [DBG] mds.? [v2:172.21.15.187:6826/4049116176,v1:172.21.15.187:6827/4049116176] up:reconnect 2021-11-20T05:36:16.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:16 smithi187 conmon[36358]: cluster 2021-11-20T05:36:15.308249+0000 mon.smithi058 (mon.0) 645 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:16.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:16 smithi187 conmon[36358]: cluster 2021-11-20T05:36:15.329992+0000 mds.cephfs.smithi187.fikisa 2021-11-20T05:36:16.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:16 smithi187 conmon[36358]: (mds.0) 1 : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0229996 2021-11-20T05:36:16.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:16 smithi058 conmon[33537]: cluster 2021-11-20T05:36:15.308168+0000 2021-11-20T05:36:16.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:16 smithi058 conmon[33537]: mon.smithi058 (mon.0) 644 : cluster [DBG] mds.? [v2:172.21.15.187:6826/4049116176,v1:172.21.15.187:6827/4049116176] up:reconnect 2021-11-20T05:36:16.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:16 smithi058 conmon[33537]: cluster 2021-11-20T05:36:15.308249+0000 2021-11-20T05:36:16.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:16 smithi058 conmon[33537]: mon.smithi058 (mon.0) 645 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:16.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:16 smithi058 conmon[33537]: cluster 2021-11-20T05:36:15.329992+0000 mds.cephfs.smithi187.fikisa (mds.0) 1 2021-11-20T05:36:16.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:16 smithi058 conmon[33537]: : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0229996 2021-11-20T05:36:17.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:17 smithi058 conmon[33537]: cluster 2021-11-20T05:36:16.256098+0000 mgr.smithi058.zagdlk (mgr.14164) 257 2021-11-20T05:36:17.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:17 smithi058 conmon[33537]: : cluster [DBG] pgmap v226: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T05:36:17.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:17 smithi058 conmon[33537]: cluster 2021-11-20T05:36:16.321271+0000 mon.smithi058 (mon.0) 646 : cluster [DBG] mds.? [v2:172.21.15.187:6826/4049116176,v1:172.21.15.187:6827/4049116176] up:rejoin 2021-11-20T05:36:17.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:17 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:36:17.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:17 smithi058 conmon[33537]: -11-20T05:36:16.321364+0000 mon.smithi058 (mon.0) 647 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:17.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:17 smithi187 conmon[36358]: cluster 2021-11-20T05:36:16.256098+0000 mgr.smithi058.zagdlk (mgr.14164) 257 : cluster [DBG] 2021-11-20T05:36:17.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:17 smithi187 conmon[36358]: pgmap v226: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T05:36:17.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:17 smithi187 conmon[36358]: cluster 2021-11-20T05:36:16.321271+0000 mon.smithi058 (mon.0) 646 : cluster [DBG] mds.? [v2:172.21.15.187:6826/4049116176,v1:172.21.15.187:6827/4049116176] up:rejoin 2021-11-20T05:36:17.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:17 smithi187 conmon[36358]: cluster 2021-11-20T05:36:16.321364+0000 mon.smithi058 (mon.0) 647 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:18.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:18 smithi187 conmon[36358]: cluster 2021-11-20T05:36:17.326338+0000 mon.smithi058 (mon.0) 648 : 2021-11-20T05:36:18.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:18 smithi187 conmon[36358]: cluster [DBG] mds.? [v2:172.21.15.58:6826/444955435,v1:172.21.15.58:6827/444955435] up:standby 2021-11-20T05:36:18.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:18 smithi187 conmon[36358]: cluster 2021-11-20T05:36:17.326430+0000 mon.smithi058 (mon.0) 649 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:18.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:18 smithi058 conmon[33537]: cluster 2021-11-20T05:36:17.326338+0000 mon.smithi058 (mon.0) 648 : cluster [DBG] 2021-11-20T05:36:18.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:18 smithi058 conmon[33537]: mds.? [v2:172.21.15.58:6826/444955435,v1:172.21.15.58:6827/444955435] up:standby 2021-11-20T05:36:18.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:18 smithi058 conmon[33537]: cluster 2021-11-20T05:36:17.326430+0000 mon.smithi058 (mon.0) 649 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:19.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:19 smithi058 conmon[33537]: cluster 2021-11-20T05:36:18.256745+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:36:19.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:19 smithi058 conmon[33537]: 258 : cluster [DBG] pgmap v227: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:19.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:19 smithi187 conmon[36358]: cluster 2021-11-20T05:36:18.256745+0000 mgr.smithi058.zagdlk (mgr.14164) 258 : cluster 2021-11-20T05:36:19.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:19 smithi187 conmon[36358]: [DBG] pgmap v227: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:21.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:21 smithi187 conmon[36358]: cluster 2021-11-20T05:36:20.257394+0000 mgr.smithi058.zagdlk (mgr.14164) 259 2021-11-20T05:36:21.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:21 smithi187 conmon[36358]: : cluster [DBG] pgmap v228: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:21.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:21 smithi058 conmon[33537]: cluster 2021-11-20T05:36:20.257394+0000 mgr.smithi058.zagdlk (mgr.14164) 259 : cluster 2021-11-20T05:36:21.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:21 smithi058 conmon[33537]: [DBG] pgmap v228: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:23.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:23 smithi187 conmon[36358]: cluster 2021-11-20T05:36:22.258101+0000 mgr.smithi058.zagdlk (mgr.14164) 260 : 2021-11-20T05:36:23.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:23 smithi187 conmon[36358]: cluster [DBG] pgmap v229: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:23.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:23 smithi058 conmon[33537]: cluster 2021-11-20T05:36:22.258101+0000 mgr.smithi058.zagdlk (mgr.14164) 260 : cluster [DBG] pgmap v229: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:25.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:25 smithi058 conmon[33537]: cluster 2021-11-20T05:36:24.258758+0000 mgr.smithi058.zagdlk (mgr.14164) 261 : cluster [DBG] pgmap v230: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T05:36:25.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:25 smithi187 conmon[36358]: cluster 2021-11-20T05:36:24.258758+0000 mgr.smithi058.zagdlk (mgr.14164) 261 : cluster [DBG] pgmap v230: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T05:36:26.645 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:26.646+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:26.645 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:26.647+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T05:36:26.646 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:26.647+0000 7f9adad9d700 1 -- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 msgr2=0x5644ba26db60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T05:36:26.646 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:26.647+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T05:36:26.647 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:26.647+0000 7f9adad9d700 1 --2- 172.21.15.58:0/2412268982 >> [v2:172.21.15.58:6826/3420124164,v1:172.21.15.58:6827/3420124164] conn(0x5644ba26b770 0x5644ba26db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T05:36:27.574 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:27 smithi187 conmon[36358]: cluster 2021-11-20T05:36:26.259341+0000 mgr.smithi058.zagdlk (mgr.14164) 262 : cluster [DBG] pgmap v231: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 0 op/s 2021-11-20T05:36:27.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:27 smithi058 conmon[33537]: cluster 2021-11-20T05:36:26.259341+0000 mgr.smithi058.zagdlk (mgr.14164) 262 : 2021-11-20T05:36:27.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:27 smithi058 conmon[33537]: cluster [DBG] pgmap v231: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 0 op/s 2021-11-20T05:36:29.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:29 smithi187 conmon[36358]: cluster 2021-11-20T05:36:28.260019+0000 mgr.smithi058.zagdlk (mgr.14164) 263 : cluster [DBG] pgmap v232: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 0 op/s 2021-11-20T05:36:29.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:29 smithi187 conmon[36358]: audit 2021-11-20T05:36:28.924179+0000 mon.smithi058 (mon.0) 650 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:36:29.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:29 smithi187 conmon[36358]: audit 2021-11-20T05:36:29.313952+0000 mon.smithi058 (mon.0) 651 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:36:29.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:29 smithi058 conmon[33537]: cluster 2021-11-20T05:36:28.260019+0000 mgr.smithi058.zagdlk 2021-11-20T05:36:29.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:29 smithi058 conmon[33537]: (mgr.14164) 263 : cluster [DBG] pgmap v232: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 0 op/s 2021-11-20T05:36:29.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:29 smithi058 conmon[33537]: audit 2021-11-20T05:36:28.924179+0000 mon.smithi058 (mon.0) 650 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:36:29.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:29 smithi058 conmon[33537]: audit 2021-11-20T05:36:29.313952+0000 mon.smithi058 (mon.0) 651 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:36:31.059 DEBUG:teuthology.orchestra.run:got remote process result: 124 2021-11-20T05:36:31.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:31 smithi187 conmon[36358]: cluster 2021-11-20T05:36:30.260479+0000 mgr.smithi058.zagdlk (mgr.14164) 264 : cluster [DBG] pgmap v233: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:36:31.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:31 smithi058 conmon[33537]: cluster 2021-11-20T05:36:30.260479+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:36:31.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:31 smithi058 conmon[33537]: 264 : cluster [DBG] pgmap v233: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:36:32.062 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:32.200 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.202+0000 7f2439f67700 1 -- 172.21.15.58:0/2265014868 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f24340f5580 msgr2=0x7f24340f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:32.201 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.202+0000 7f2439f67700 1 --2- 172.21.15.58:0/2265014868 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f24340f5580 0x7f24340f59a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f2424004660 tx=0x7f2424009b30).stop 2021-11-20T05:36:32.201 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.202+0000 7f2439f67700 1 -- 172.21.15.58:0/2265014868 shutdown_connections 2021-11-20T05:36:32.201 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.202+0000 7f2439f67700 1 --2- 172.21.15.58:0/2265014868 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24340f6820 0x7f24340f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:32.202 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.202+0000 7f2439f67700 1 --2- 172.21.15.58:0/2265014868 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f24340f5580 0x7f24340f59a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:32.202 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.202+0000 7f2439f67700 1 -- 172.21.15.58:0/2265014868 >> 172.21.15.58:0/2265014868 conn(0x7f24340f0ce0 msgr2=0x7f24340f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:32.202 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.202+0000 7f2439f67700 1 -- 172.21.15.58:0/2265014868 shutdown_connections 2021-11-20T05:36:32.202 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.202+0000 7f2439f67700 1 -- 172.21.15.58:0/2265014868 wait complete. 2021-11-20T05:36:32.203 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.203+0000 7f2439f67700 1 Processor -- start 2021-11-20T05:36:32.205 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.203+0000 7f2439f67700 1 -- start start 2021-11-20T05:36:32.205 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.204+0000 7f2439f67700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24340f5580 0x7f24340604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:32.206 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.204+0000 7f2439f67700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f24340f6820 0x7f2434109a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:32.206 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.204+0000 7f2439f67700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2434109f80 con 0x7f24340f6820 2021-11-20T05:36:32.206 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.204+0000 7f2439f67700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243410a0c0 con 0x7f24340f5580 2021-11-20T05:36:32.206 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.204+0000 7f2432ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f24340f6820 0x7f2434109a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:32.207 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.204+0000 7f2432ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f24340f6820 0x7f2434109a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41134/0 (socket says 172.21.15.58:41134) 2021-11-20T05:36:32.207 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.204+0000 7f2432ffd700 1 -- 172.21.15.58:0/3340684481 learned_addr learned my addr 172.21.15.58:0/3340684481 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:32.207 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.204+0000 7f24337fe700 1 --2- 172.21.15.58:0/3340684481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24340f5580 0x7f24340604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:32.207 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.204+0000 7f2432ffd700 1 -- 172.21.15.58:0/3340684481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24340f5580 msgr2=0x7f24340604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:32.208 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.204+0000 7f2432ffd700 1 --2- 172.21.15.58:0/3340684481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24340f5580 0x7f24340604f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:32.208 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.204+0000 7f2432ffd700 1 -- 172.21.15.58:0/3340684481 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2424005040 con 0x7f24340f6820 2021-11-20T05:36:32.208 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.204+0000 7f2432ffd700 1 --2- 172.21.15.58:0/3340684481 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f24340f6820 0x7f2434109a90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f242800c530 tx=0x7f2428007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:32.208 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.204+0000 7f241bfff700 1 -- 172.21.15.58:0/3340684481 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f242800f650 con 0x7f24340f6820 2021-11-20T05:36:32.208 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.204+0000 7f241bfff700 1 -- 172.21.15.58:0/3340684481 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f242800fc60 con 0x7f24340f6820 2021-11-20T05:36:32.209 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.204+0000 7f2439f67700 1 -- 172.21.15.58:0/3340684481 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f243410a3a0 con 0x7f24340f6820 2021-11-20T05:36:32.209 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.204+0000 7f241bfff700 1 -- 172.21.15.58:0/3340684481 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2428012c80 con 0x7f24340f6820 2021-11-20T05:36:32.209 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.204+0000 7f2439f67700 1 -- 172.21.15.58:0/3340684481 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f243410a960 con 0x7f24340f6820 2021-11-20T05:36:32.209 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.205+0000 7f2439f67700 1 -- 172.21.15.58:0/3340684481 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2434027a00 con 0x7f24340f6820 2021-11-20T05:36:32.210 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.206+0000 7f241bfff700 1 -- 172.21.15.58:0/3340684481 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2428010070 con 0x7f24340f6820 2021-11-20T05:36:32.210 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.206+0000 7f241bfff700 1 --2- 172.21.15.58:0/3340684481 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f241c05c2a0 0x7f241c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:32.210 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.206+0000 7f241bfff700 1 -- 172.21.15.58:0/3340684481 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f2428075eb0 con 0x7f24340f6820 2021-11-20T05:36:32.210 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.206+0000 7f24337fe700 1 --2- 172.21.15.58:0/3340684481 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f241c05c2a0 0x7f241c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:32.210 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.207+0000 7f24337fe700 1 --2- 172.21.15.58:0/3340684481 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f241c05c2a0 0x7f241c05e760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f2424009d60 tx=0x7f2424009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:32.211 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.210+0000 7f241bfff700 1 -- 172.21.15.58:0/3340684481 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f24280452b0 con 0x7f24340f6820 2021-11-20T05:36:32.410 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.413+0000 7f2439f67700 1 -- 172.21.15.58:0/3340684481 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24340fdd10 con 0x7f24340f6820 2021-11-20T05:36:32.411 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.413+0000 7f241bfff700 1 -- 172.21.15.58:0/3340684481 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+2945 (secure 0 0 0) 0x7f242804c2f0 con 0x7f24340f6820 2021-11-20T05:36:32.412 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:32.412 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":45},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:32.326604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14572,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14572":{"gid":14572,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":53,"state":"up:replay","state_seq":2,"addr":"172.21.15.58:6827/444955435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":444955435},{"type":"v1","addr":"172.21.15.58:6827","nonce":444955435}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:32.413 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.415+0000 7f2439f67700 1 -- 172.21.15.58:0/3340684481 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f241c05c2a0 msgr2=0x7f241c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:32.413 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.415+0000 7f2439f67700 1 --2- 172.21.15.58:0/3340684481 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f241c05c2a0 0x7f241c05e760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f2424009d60 tx=0x7f2424009b30).stop 2021-11-20T05:36:32.414 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.415+0000 7f2439f67700 1 -- 172.21.15.58:0/3340684481 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f24340f6820 msgr2=0x7f2434109a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:32.414 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.415+0000 7f2439f67700 1 --2- 172.21.15.58:0/3340684481 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f24340f6820 0x7f2434109a90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f242800c530 tx=0x7f2428007e20).stop 2021-11-20T05:36:32.414 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.415+0000 7f2439f67700 1 -- 172.21.15.58:0/3340684481 shutdown_connections 2021-11-20T05:36:32.414 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.415+0000 7f2439f67700 1 --2- 172.21.15.58:0/3340684481 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f241c05c2a0 0x7f241c05e760 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:32.415 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.415+0000 7f2439f67700 1 --2- 172.21.15.58:0/3340684481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f24340f5580 0x7f24340604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:32.415 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.415+0000 7f2439f67700 1 --2- 172.21.15.58:0/3340684481 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f24340f6820 0x7f2434109a90 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:32.415 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.415+0000 7f2439f67700 1 -- 172.21.15.58:0/3340684481 >> 172.21.15.58:0/3340684481 conn(0x7f24340f0ce0 msgr2=0x7f24340fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:32.415 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.415+0000 7f2439f67700 1 -- 172.21.15.58:0/3340684481 shutdown_connections 2021-11-20T05:36:32.415 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:32.415+0000 7f2439f67700 1 -- 172.21.15.58:0/3340684481 wait complete. 2021-11-20T05:36:32.416 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 53 2021-11-20T05:36:32.423 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:32 smithi058 conmon[33537]: cluster 2021-11-20T05:36:31.494223+0000 mon.smithi058 (mon.0) 652 : cluster [INF] daemon mds.cephfs.smithi187.fikisa restarted 2021-11-20T05:36:32.423 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:32 smithi058 conmon[33537]: cluster 2021-11-20T05:36:31.499433+0000 mon.smithi058 (mon.0) 653 : cluster 2021-11-20T05:36:32.424 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:32 smithi058 conmon[33537]: [DBG] osdmap e50: 6 total, 6 up, 6 in 2021-11-20T05:36:32.425 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:32.326604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14572, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14572': {'gid': 14572, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 53, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.58:6827/444955435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 444955435}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 444955435}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:32.425 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14572:up:replay 2021-11-20T05:36:32.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:32 smithi187 conmon[36358]: cluster 2021-11-20T05:36:31.494223+0000 mon.smithi058 (mon.0) 652 : cluster [INF] daemon mds.cephfs.smithi187.fikisa restarted 2021-11-20T05:36:32.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:32 smithi187 conmon[36358]: cluster 2021-11-20T05:36:31.499433+0000 mon.smithi058 2021-11-20T05:36:32.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:32 smithi187 conmon[36358]: (mon.0) 653 : cluster [DBG] osdmap e50: 6 total, 6 up, 6 in 2021-11-20T05:36:33.427 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:33.564 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.562+0000 7f2fdcf77700 1 -- 172.21.15.58:0/1977254110 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2fd80f4d40 msgr2=0x7f2fd80f5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:33.564 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.562+0000 7f2fdcf77700 1 --2- 172.21.15.58:0/1977254110 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2fd80f4d40 0x7f2fd80f5160 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f2fc8004660 tx=0x7f2fc8009b30).stop 2021-11-20T05:36:33.564 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.562+0000 7f2fdcf77700 1 -- 172.21.15.58:0/1977254110 shutdown_connections 2021-11-20T05:36:33.565 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.562+0000 7f2fdcf77700 1 --2- 172.21.15.58:0/1977254110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fd80f5fe0 0x7f2fd80f6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:33.565 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.562+0000 7f2fdcf77700 1 --2- 172.21.15.58:0/1977254110 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2fd80f4d40 0x7f2fd80f5160 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:33.565 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.562+0000 7f2fdcf77700 1 -- 172.21.15.58:0/1977254110 >> 172.21.15.58:0/1977254110 conn(0x7f2fd80f04a0 msgr2=0x7f2fd80f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:33.565 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.562+0000 7f2fdcf77700 1 -- 172.21.15.58:0/1977254110 shutdown_connections 2021-11-20T05:36:33.566 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.562+0000 7f2fdcf77700 1 -- 172.21.15.58:0/1977254110 wait complete. 2021-11-20T05:36:33.566 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.562+0000 7f2fdcf77700 1 Processor -- start 2021-11-20T05:36:33.566 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.562+0000 7f2fdcf77700 1 -- start start 2021-11-20T05:36:33.567 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.562+0000 7f2fdcf77700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2fd80f4d40 0x7f2fd8106f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:33.567 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.562+0000 7f2fdcf77700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fd80f5fe0 0x7f2fd8107400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:33.567 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.562+0000 7f2fdcf77700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd81079b0 con 0x7f2fd80f4d40 2021-11-20T05:36:33.567 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.562+0000 7f2fdcf77700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd8107af0 con 0x7f2fd80f5fe0 2021-11-20T05:36:33.568 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.563+0000 7f2fd659c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2fd80f4d40 0x7f2fd8106f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:33.568 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.563+0000 7f2fd659c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2fd80f4d40 0x7f2fd8106f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41154/0 (socket says 172.21.15.58:41154) 2021-11-20T05:36:33.568 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.563+0000 7f2fd659c700 1 -- 172.21.15.58:0/225143710 learned_addr learned my addr 172.21.15.58:0/225143710 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:33.568 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.563+0000 7f2fd5d9b700 1 --2- 172.21.15.58:0/225143710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fd80f5fe0 0x7f2fd8107400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:33.569 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.563+0000 7f2fd659c700 1 -- 172.21.15.58:0/225143710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fd80f5fe0 msgr2=0x7f2fd8107400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:33.569 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.563+0000 7f2fd659c700 1 --2- 172.21.15.58:0/225143710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fd80f5fe0 0x7f2fd8107400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:33.569 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.563+0000 7f2fd659c700 1 -- 172.21.15.58:0/225143710 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc8005040 con 0x7f2fd80f4d40 2021-11-20T05:36:33.569 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.563+0000 7f2fd659c700 1 --2- 172.21.15.58:0/225143710 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2fd80f4d40 0x7f2fd8106f10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f2fc8007a60 tx=0x7f2fc8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:33.570 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.563+0000 7f2fc6ffd700 1 -- 172.21.15.58:0/225143710 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fc8018070 con 0x7f2fd80f4d40 2021-11-20T05:36:33.570 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.563+0000 7f2fdcf77700 1 -- 172.21.15.58:0/225143710 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd810c530 con 0x7f2fd80f4d40 2021-11-20T05:36:33.570 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.563+0000 7f2fc6ffd700 1 -- 172.21.15.58:0/225143710 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fc8013410 con 0x7f2fd80f4d40 2021-11-20T05:36:33.570 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.563+0000 7f2fc6ffd700 1 -- 172.21.15.58:0/225143710 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fc801c620 con 0x7f2fd80f4d40 2021-11-20T05:36:33.571 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.563+0000 7f2fdcf77700 1 -- 172.21.15.58:0/225143710 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd810caf0 con 0x7f2fd80f4d40 2021-11-20T05:36:33.571 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.564+0000 7f2fdcf77700 1 -- 172.21.15.58:0/225143710 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd8027a00 con 0x7f2fd80f4d40 2021-11-20T05:36:33.571 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.565+0000 7f2fc6ffd700 1 -- 172.21.15.58:0/225143710 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2fc801c890 con 0x7f2fd80f4d40 2021-11-20T05:36:33.571 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.565+0000 7f2fc6ffd700 1 --2- 172.21.15.58:0/225143710 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2fc005c2a0 0x7f2fc005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:33.571 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.565+0000 7f2fc6ffd700 1 -- 172.21.15.58:0/225143710 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f2fc8078060 con 0x7f2fd80f4d40 2021-11-20T05:36:33.572 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.565+0000 7f2fd5d9b700 1 --2- 172.21.15.58:0/225143710 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2fc005c2a0 0x7f2fc005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:33.572 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.566+0000 7f2fd5d9b700 1 --2- 172.21.15.58:0/225143710 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2fc005c2a0 0x7f2fc005e760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f2fcc004a70 tx=0x7f2fcc00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:33.573 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.569+0000 7f2fc6ffd700 1 -- 172.21.15.58:0/225143710 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2fc804aa70 con 0x7f2fd80f4d40 2021-11-20T05:36:33.576 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: cluster 2021-11-20T05:36:32.260885+0000 mgr.smithi058.zagdlk (mgr.14164) 265 : cluster [DBG] pgmap v235: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:36:33.576 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: cluster 2021-11-20T05:36:32.326518+0000 2021-11-20T05:36:33.576 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: mon.smithi058 (mon.0) 654 : cluster [DBG] mds.? [v2:172.21.15.187:6826/2249219455,v1:172.21.15.187:6827/2249219455] up:boot 2021-11-20T05:36:33.576 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: cluster 2021-11-20T05:36:32.326581+0000 mon.smithi058 (mon.0) 655 : cluster [INF] Standby daemon mds.cephfs.smithi058.rxapzd assigned to filesystem cephfs as rank 0 2021-11-20T05:36:33.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: cluster 2021-11-20T05:36:33.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: 2021-11-20T05:36:32.326785+0000 mon.smithi058 (mon.0) 656 : 2021-11-20T05:36:33.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 3 up:standby, 1 failed 2021-11-20T05:36:33.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: audit 2021-11 2021-11-20T05:36:33.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: -20T05:36:32.326899+0000 mon.smithi058 ( 2021-11-20T05:36:33.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: mon.0) 657 : audit [DBG] 2021-11-20T05:36:33.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.fikisa"}]: dispatch 2021-11-20T05:36:33.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: cluster 2021- 2021-11-20T05:36:33.579 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: 11-20T05:36:32.329012 2021-11-20T05:36:33.579 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 658 : 2021-11-20T05:36:33.579 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:33.579 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: audit 2021-11 2021-11-20T05:36:33.580 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: -20T05:36:32.414352+0000 mon.smithi058 2021-11-20T05:36:33.580 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: (mon.0) 659 : audit [DBG] 2021-11-20T05:36:33.580 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:33 smithi187 conmon[36358]: from='client.? 172.21.15.58:0/3340684481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:33.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: cluster 2021-11-20T05:36:32.260885+0000 mgr.smithi058.zagdlk (mgr.14164) 265 2021-11-20T05:36:33.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: : cluster [DBG] pgmap v235: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:36:33.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: cluster 2021-11-20T05:36:32.326518+0000 mon.smithi058 (mon.0) 654 : cluster [DBG] mds.? [v2:172.21.15.187:6826/2249219455,v1:172.21.15.187:6827/2249219455] up:boot 2021-11-20T05:36:33.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: cluster 2021- 2021-11-20T05:36:33.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: 11-20T05:36:32.326581+0000 mon.smithi058 ( 2021-11-20T05:36:33.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: mon.0) 655 : cluster [INF] 2021-11-20T05:36:33.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: Standby daemon mds.cephfs.smithi058.rxapzd assigned to filesystem cephfs as rank 0 2021-11-20T05:36:33.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:36:33.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: -11-20T05:36:32. 2021-11-20T05:36:33.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: 326785+0000 mon.smithi058 (mon.0) 2021-11-20T05:36:33.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: 656 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 3 up:standby, 1 failed 2021-11-20T05:36:33.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: audit 2021-11-20T05:36:33.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: 2021-11-20T05:36:32.326899+0000 2021-11-20T05:36:33.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: mon.smithi058 (mon.0) 657 : 2021-11-20T05:36:33.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.fikisa"}]: dispatch 2021-11-20T05:36:33.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:36:33.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: -11-20T05:36:32. 2021-11-20T05:36:33.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: 329012+0000 mon.smithi058 (mon.0 2021-11-20T05:36:33.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: ) 658 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:33.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: 2021-11-20T05:36:33.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: audit 2021 2021-11-20T05:36:33.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: -11-20T05:36:32 2021-11-20T05:36:33.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: .414352+0000 mon.smithi058 (mon. 2021-11-20T05:36:33.724 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: 0) 659 : audit [DBG] from='client.? 172.21.15.58:0/3340684481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:33.724 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:33 smithi058 conmon[33537]: 2021-11-20T05:36:33.761 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.763+0000 7f2fdcf77700 1 -- 172.21.15.58:0/225143710 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fd80fd4d0 con 0x7f2fd80f4d40 2021-11-20T05:36:33.762 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.764+0000 7f2fc6ffd700 1 -- 172.21.15.58:0/225143710 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2946 (secure 0 0 0) 0x7f2fc804e090 con 0x7f2fd80f4d40 2021-11-20T05:36:33.762 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:33.763 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":45},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:33.330439+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14572,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14572":{"gid":14572,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":53,"state":"up:resolve","state_seq":6,"addr":"172.21.15.58:6827/444955435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":444955435},{"type":"v1","addr":"172.21.15.58:6827","nonce":444955435}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:33.764 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.766+0000 7f2fdcf77700 1 -- 172.21.15.58:0/225143710 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2fc005c2a0 msgr2=0x7f2fc005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:33.764 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.766+0000 7f2fdcf77700 1 --2- 172.21.15.58:0/225143710 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2fc005c2a0 0x7f2fc005e760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f2fcc004a70 tx=0x7f2fcc00b040).stop 2021-11-20T05:36:33.765 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.766+0000 7f2fdcf77700 1 -- 172.21.15.58:0/225143710 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2fd80f4d40 msgr2=0x7f2fd8106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:33.765 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.766+0000 7f2fdcf77700 1 --2- 172.21.15.58:0/225143710 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2fd80f4d40 0x7f2fd8106f10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f2fc8007a60 tx=0x7f2fc8009b30).stop 2021-11-20T05:36:33.765 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.766+0000 7f2fdcf77700 1 -- 172.21.15.58:0/225143710 shutdown_connections 2021-11-20T05:36:33.765 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.766+0000 7f2fdcf77700 1 --2- 172.21.15.58:0/225143710 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2fc005c2a0 0x7f2fc005e760 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:33.765 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.766+0000 7f2fdcf77700 1 --2- 172.21.15.58:0/225143710 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2fd80f4d40 0x7f2fd8106f10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:33.766 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.766+0000 7f2fdcf77700 1 --2- 172.21.15.58:0/225143710 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2fd80f5fe0 0x7f2fd8107400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:33.766 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.766+0000 7f2fdcf77700 1 -- 172.21.15.58:0/225143710 >> 172.21.15.58:0/225143710 conn(0x7f2fd80f04a0 msgr2=0x7f2fd80fa520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:33.766 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.766+0000 7f2fdcf77700 1 -- 172.21.15.58:0/225143710 shutdown_connections 2021-11-20T05:36:33.766 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:33.766+0000 7f2fdcf77700 1 -- 172.21.15.58:0/225143710 wait complete. 2021-11-20T05:36:33.767 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 54 2021-11-20T05:36:33.775 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 54, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:33.330439+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14572, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14572': {'gid': 14572, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 53, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.58:6827/444955435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 444955435}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 444955435}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:33.776 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14572:up:resolve 2021-11-20T05:36:34.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:34 smithi187 conmon[36358]: cluster 2021-11-20T05:36:33.333462+0000 mon.smithi058 (mon.0) 660 : cluster [DBG] mds.? [v2:172.21.15.58:6826/444955435,v1:172.21.15.58:6827/444955435] up:resolve 2021-11-20T05:36:34.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:34 smithi187 conmon[36358]: cluster 2021-11-20T05:36:33.333558+0000 mon.smithi058 (mon.0) 661 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:34.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:34 smithi187 conmon[36358]: audit 2021-11-20T05:36:33.765047+0000 mon.smithi058 (mon.0) 2021-11-20T05:36:34.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:34 smithi187 conmon[36358]: 662 : audit [DBG] from='client.? 172.21.15.58:0/225143710' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:34.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:34 smithi058 conmon[33537]: cluster 2021-11-20T05:36:33.333462+0000 mon.smithi058 (mon 2021-11-20T05:36:34.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:34 smithi058 conmon[33537]: .0) 660 : cluster [DBG] mds.? [v2:172.21.15.58:6826/444955435,v1:172.21.15.58:6827/444955435] up:resolve 2021-11-20T05:36:34.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:34 smithi058 conmon[33537]: cluster 2021-11-20T05:36:33.333558+0000 mon.smithi058 (mon.0) 2021-11-20T05:36:34.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:34 smithi058 conmon[33537]: 661 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:34.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:34 smithi058 conmon[33537]: audit 2021-11-20T05: 2021-11-20T05:36:34.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:34 smithi058 conmon[33537]: 36:33.765047+0000 mon.smithi058 ( 2021-11-20T05:36:34.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:34 smithi058 conmon[33537]: mon.0) 662 : audit [DBG] from='client.? 172.21.15.58:0/225143710' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:34.777 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:34.912 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.912+0000 7f9e52d2f700 1 -- 172.21.15.58:0/1526755624 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e4c0f6860 msgr2=0x7f9e4c0f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:34.913 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.912+0000 7f9e52d2f700 1 --2- 172.21.15.58:0/1526755624 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e4c0f6860 0x7f9e4c0f6cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f9e38004660 tx=0x7f9e38009b30).stop 2021-11-20T05:36:34.913 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.912+0000 7f9e52d2f700 1 -- 172.21.15.58:0/1526755624 shutdown_connections 2021-11-20T05:36:34.913 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.912+0000 7f9e52d2f700 1 --2- 172.21.15.58:0/1526755624 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e4c0f6860 0x7f9e4c0f6cc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:34.913 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.912+0000 7f9e52d2f700 1 --2- 172.21.15.58:0/1526755624 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e4c0f55c0 0x7f9e4c0f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:34.914 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.912+0000 7f9e52d2f700 1 -- 172.21.15.58:0/1526755624 >> 172.21.15.58:0/1526755624 conn(0x7f9e4c0f0d40 msgr2=0x7f9e4c0f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:34.914 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.912+0000 7f9e52d2f700 1 -- 172.21.15.58:0/1526755624 shutdown_connections 2021-11-20T05:36:34.914 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.912+0000 7f9e52d2f700 1 -- 172.21.15.58:0/1526755624 wait complete. 2021-11-20T05:36:34.914 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.913+0000 7f9e52d2f700 1 Processor -- start 2021-11-20T05:36:34.915 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.913+0000 7f9e52d2f700 1 -- start start 2021-11-20T05:36:34.915 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.913+0000 7f9e52d2f700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e4c0f55c0 0x7f9e4c105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:34.915 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.913+0000 7f9e52d2f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e4c0f6860 0x7f9e4c106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:34.915 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.913+0000 7f9e52d2f700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c106840 con 0x7f9e4c0f55c0 2021-11-20T05:36:34.915 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.913+0000 7f9e52d2f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c106980 con 0x7f9e4c0f6860 2021-11-20T05:36:34.916 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.913+0000 7f9e50acb700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e4c0f55c0 0x7f9e4c105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:34.916 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.913+0000 7f9e50acb700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e4c0f55c0 0x7f9e4c105da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41166/0 (socket says 172.21.15.58:41166) 2021-11-20T05:36:34.916 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.913+0000 7f9e50acb700 1 -- 172.21.15.58:0/2451423350 learned_addr learned my addr 172.21.15.58:0/2451423350 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:34.916 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.913+0000 7f9e4bfff700 1 --2- 172.21.15.58:0/2451423350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e4c0f6860 0x7f9e4c106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:34.917 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.914+0000 7f9e50acb700 1 -- 172.21.15.58:0/2451423350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e4c0f6860 msgr2=0x7f9e4c106290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:34.917 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.914+0000 7f9e50acb700 1 --2- 172.21.15.58:0/2451423350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e4c0f6860 0x7f9e4c106290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:34.917 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.914+0000 7f9e50acb700 1 -- 172.21.15.58:0/2451423350 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e38005040 con 0x7f9e4c0f55c0 2021-11-20T05:36:34.917 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.914+0000 7f9e4bfff700 1 --2- 172.21.15.58:0/2451423350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e4c0f6860 0x7f9e4c106290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:36:34.918 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.914+0000 7f9e50acb700 1 --2- 172.21.15.58:0/2451423350 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e4c0f55c0 0x7f9e4c105da0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f9e4000f910 tx=0x7f9e40009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:34.918 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.914+0000 7f9e497fa700 1 -- 172.21.15.58:0/2451423350 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e40009500 con 0x7f9e4c0f55c0 2021-11-20T05:36:34.918 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.914+0000 7f9e497fa700 1 -- 172.21.15.58:0/2451423350 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e40009660 con 0x7f9e4c0f55c0 2021-11-20T05:36:34.918 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.914+0000 7f9e52d2f700 1 -- 172.21.15.58:0/2451423350 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4c05fa80 con 0x7f9e4c0f55c0 2021-11-20T05:36:34.919 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.914+0000 7f9e497fa700 1 -- 172.21.15.58:0/2451423350 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e4000de20 con 0x7f9e4c0f55c0 2021-11-20T05:36:34.919 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.914+0000 7f9e52d2f700 1 -- 172.21.15.58:0/2451423350 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e4c060040 con 0x7f9e4c0f55c0 2021-11-20T05:36:34.919 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.915+0000 7f9e52d2f700 1 -- 172.21.15.58:0/2451423350 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e4c0fdd50 con 0x7f9e4c0f55c0 2021-11-20T05:36:34.920 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.915+0000 7f9e497fa700 1 -- 172.21.15.58:0/2451423350 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9e4000e040 con 0x7f9e4c0f55c0 2021-11-20T05:36:34.920 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.916+0000 7f9e497fa700 1 --2- 172.21.15.58:0/2451423350 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9e3c05c2a0 0x7f9e3c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:34.921 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.916+0000 7f9e497fa700 1 -- 172.21.15.58:0/2451423350 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f9e40078a80 con 0x7f9e4c0f55c0 2021-11-20T05:36:34.921 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.916+0000 7f9e4bfff700 1 --2- 172.21.15.58:0/2451423350 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9e3c05c2a0 0x7f9e3c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:34.921 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.916+0000 7f9e4bfff700 1 --2- 172.21.15.58:0/2451423350 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9e3c05c2a0 0x7f9e3c05e760 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f9e38005010 tx=0x7f9e38009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:34.921 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:34.920+0000 7f9e497fa700 1 -- 172.21.15.58:0/2451423350 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e4004b490 con 0x7f9e4c0f55c0 2021-11-20T05:36:35.112 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:35.115+0000 7f9e52d2f700 1 -- 172.21.15.58:0/2451423350 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e4c10b3c0 con 0x7f9e4c0f55c0 2021-11-20T05:36:35.113 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:35.115+0000 7f9e497fa700 1 -- 172.21.15.58:0/2451423350 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+2948 (secure 0 0 0) 0x7f9e4004eab0 con 0x7f9e4c0f55c0 2021-11-20T05:36:35.113 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:35.114 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":45},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":52}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:34.331355+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14572,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14572":{"gid":14572,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":53,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.58:6827/444955435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":444955435},{"type":"v1","addr":"172.21.15.58:6827","nonce":444955435}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:35.115 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:35.117+0000 7f9e52d2f700 1 -- 172.21.15.58:0/2451423350 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9e3c05c2a0 msgr2=0x7f9e3c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:35.115 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:35.117+0000 7f9e52d2f700 1 --2- 172.21.15.58:0/2451423350 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9e3c05c2a0 0x7f9e3c05e760 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f9e38005010 tx=0x7f9e38009b30).stop 2021-11-20T05:36:35.115 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:35.117+0000 7f9e52d2f700 1 -- 172.21.15.58:0/2451423350 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e4c0f55c0 msgr2=0x7f9e4c105da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:35.116 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:35.117+0000 7f9e52d2f700 1 --2- 172.21.15.58:0/2451423350 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e4c0f55c0 0x7f9e4c105da0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f9e4000f910 tx=0x7f9e40009d80).stop 2021-11-20T05:36:35.116 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:35.117+0000 7f9e52d2f700 1 -- 172.21.15.58:0/2451423350 shutdown_connections 2021-11-20T05:36:35.116 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:35.117+0000 7f9e52d2f700 1 --2- 172.21.15.58:0/2451423350 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9e3c05c2a0 0x7f9e3c05e760 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:35.116 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:35.117+0000 7f9e52d2f700 1 --2- 172.21.15.58:0/2451423350 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9e4c0f55c0 0x7f9e4c105da0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:35.117 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:35.117+0000 7f9e52d2f700 1 --2- 172.21.15.58:0/2451423350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9e4c0f6860 0x7f9e4c106290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:35.117 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:35.117+0000 7f9e52d2f700 1 -- 172.21.15.58:0/2451423350 >> 172.21.15.58:0/2451423350 conn(0x7f9e4c0f0d40 msgr2=0x7f9e4c0fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:35.117 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:35.117+0000 7f9e52d2f700 1 -- 172.21.15.58:0/2451423350 shutdown_connections 2021-11-20T05:36:35.117 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:35.117+0000 7f9e52d2f700 1 -- 172.21.15.58:0/2451423350 wait complete. 2021-11-20T05:36:35.118 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 55 2021-11-20T05:36:35.127 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:34.331355+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14572, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14572': {'gid': 14572, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 53, 'state': 'up:reconnect', 'state_seq': 7, 'addr': '172.21.15.58:6827/444955435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 444955435}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 444955435}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:35.127 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14572:up:reconnect 2021-11-20T05:36:35.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:35 smithi058 conmon[33537]: cluster 2021-11-20T05:36:34.261351+0000 2021-11-20T05:36:35.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:35 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 266 : cluster [DBG] pgmap v236: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:36:35.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:35 smithi058 conmon[33537]: cluster 2021-11-20T05:36:34.334639+0000 2021-11-20T05:36:35.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:35 smithi058 conmon[33537]: mon.smithi058 (mon.0) 663 : cluster [DBG] mds.? [v2:172.21.15.58:6826/444955435,v1:172.21.15.58:6827/444955435] up:reconnect 2021-11-20T05:36:35.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:35 smithi058 conmon[33537]: cluster 2021-11-20T05:36:34.334710 2021-11-20T05:36:35.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:35 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 664 : cluster [DBG] 2021-11-20T05:36:35.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:35 smithi058 conmon[33537]: fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:35.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:35 smithi058 conmon[33537]: audit 2021-11-20T05:36 2021-11-20T05:36:35.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:35 smithi058 conmon[33537]: :35.116339+0000 mon.smithi058 (mon.0) 2021-11-20T05:36:35.470 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:35 smithi058 conmon[33537]: 665 : audit [DBG] from='client.? 172.21.15.58:0/2451423350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:35.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:35 smithi187 conmon[36358]: cluster 2021-11-20T05:36:34.261351+0000 mgr.smithi058.zagdlk (mgr.14164) 266 : 2021-11-20T05:36:35.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:35 smithi187 conmon[36358]: cluster [DBG] pgmap v236: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:36:35.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:35 smithi187 conmon[36358]: cluster 2021-11-20T05:36:34.334639+0000 mon.smithi058 (mon.0) 663 : cluster [DBG] mds.? [v2:172.21.15.58:6826/444955435,v1:172.21.15.58:6827/444955435] up:reconnect 2021-11-20T05:36:35.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:35 smithi187 conmon[36358]: cluster 2021-11 2021-11-20T05:36:35.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:35 smithi187 conmon[36358]: -20T05:36:34.334710+0000 2021-11-20T05:36:35.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:35 smithi187 conmon[36358]: mon.smithi058 (mon.0) 664 : cluster [DBG] 2021-11-20T05:36:35.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:35 smithi187 conmon[36358]: fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:35.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:35 smithi187 conmon[36358]: audit 2021 2021-11-20T05:36:35.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:35 smithi187 conmon[36358]: -11-20T05:36:35 2021-11-20T05:36:35.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:35 smithi187 conmon[36358]: .116339+0000 mon.smithi058 (mon.0) 2021-11-20T05:36:35.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:35 smithi187 conmon[36358]: 665 : audit [DBG] from='client.? 172.21.15.58:0/2451423350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:36.130 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:36.273 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.274+0000 7f7044d4b700 1 -- 172.21.15.58:0/663496439 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f6820 msgr2=0x7f70400f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:36.274 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.274+0000 7f7044d4b700 1 --2- 172.21.15.58:0/663496439 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f6820 0x7f70400f6c80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f7034004660 tx=0x7f7034009b30).stop 2021-11-20T05:36:36.274 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.274+0000 7f7044d4b700 1 -- 172.21.15.58:0/663496439 shutdown_connections 2021-11-20T05:36:36.274 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.274+0000 7f7044d4b700 1 --2- 172.21.15.58:0/663496439 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f6820 0x7f70400f6c80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:36.274 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.274+0000 7f7044d4b700 1 --2- 172.21.15.58:0/663496439 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70400f5580 0x7f70400f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:36:36.275 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.274+0000 7f7044d4b700 1 -- 172.21.15.58:0/663496439 >> 172.21.15.58:0/663496439 conn(0x7f70400f0ce0 msgr2=0x7f70400f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:36.275 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.274+0000 7f7044d4b700 1 -- 172.21.15.58:0/663496439 shutdown_connections 2021-11-20T05:36:36.275 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.274+0000 7f7044d4b700 1 -- 172.21.15.58:0/663496439 wait complete. 2021-11-20T05:36:36.275 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.274+0000 7f7044d4b700 1 Processor -- start 2021-11-20T05:36:36.275 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.274+0000 7f7044d4b700 1 -- start start 2021-11-20T05:36:36.276 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.274+0000 7f7044d4b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f5580 0x7f70400604d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:36.276 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.274+0000 7f7044d4b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70400f6820 0x7f704010e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:36.276 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.274+0000 7f7044d4b700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7040107a90 con 0x7f70400f5580 2021-11-20T05:36:36.276 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.274+0000 7f7044d4b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704010e5a0 con 0x7f70400f6820 2021-11-20T05:36:36.277 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.275+0000 7f703e59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f5580 0x7f70400604d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:36.277 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.275+0000 7f703e59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f5580 0x7f70400604d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41184/0 (socket says 172.21.15.58:41184) 2021-11-20T05:36:36.277 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.275+0000 7f703e59c700 1 -- 172.21.15.58:0/2264825522 learned_addr learned my addr 172.21.15.58:0/2264825522 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:36.277 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.275+0000 7f703dd9b700 1 --2- 172.21.15.58:0/2264825522 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70400f6820 0x7f704010e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:36.277 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.275+0000 7f703e59c700 1 -- 172.21.15.58:0/2264825522 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70400f6820 msgr2=0x7f704010e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:36.278 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.275+0000 7f703e59c700 1 --2- 172.21.15.58:0/2264825522 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70400f6820 0x7f704010e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:36.278 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.275+0000 7f703e59c700 1 -- 172.21.15.58:0/2264825522 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7034005040 con 0x7f70400f5580 2021-11-20T05:36:36.278 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.275+0000 7f703e59c700 1 --2- 172.21.15.58:0/2264825522 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f5580 0x7f70400604d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f70300093f0 tx=0x7f7030006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:36.278 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.275+0000 7f702effd700 1 -- 172.21.15.58:0/2264825522 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7030009ea0 con 0x7f70400f5580 2021-11-20T05:36:36.279 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.275+0000 7f702effd700 1 -- 172.21.15.58:0/2264825522 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7030013af0 con 0x7f70400f5580 2021-11-20T05:36:36.279 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.275+0000 7f7044d4b700 1 -- 172.21.15.58:0/2264825522 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f704010e800 con 0x7f70400f5580 2021-11-20T05:36:36.279 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.275+0000 7f702effd700 1 -- 172.21.15.58:0/2264825522 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f703000dad0 con 0x7f70400f5580 2021-11-20T05:36:36.279 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.275+0000 7f7044d4b700 1 -- 172.21.15.58:0/2264825522 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f704010ed90 con 0x7f70400f5580 2021-11-20T05:36:36.279 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.276+0000 7f7044d4b700 1 -- 172.21.15.58:0/2264825522 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7040046330 con 0x7f70400f5580 2021-11-20T05:36:36.281 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.276+0000 7f702effd700 1 -- 172.21.15.58:0/2264825522 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f703001a060 con 0x7f70400f5580 2021-11-20T05:36:36.281 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.277+0000 7f702effd700 1 --2- 172.21.15.58:0/2264825522 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f702805c2a0 0x7f702805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:36.281 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.277+0000 7f702effd700 1 -- 172.21.15.58:0/2264825522 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f70300760f0 con 0x7f70400f5580 2021-11-20T05:36:36.281 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.277+0000 7f703dd9b700 1 --2- 172.21.15.58:0/2264825522 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f702805c2a0 0x7f702805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:36.282 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.277+0000 7f703dd9b700 1 --2- 172.21.15.58:0/2264825522 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f702805c2a0 0x7f702805e760 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f7034013580 tx=0x7f7034009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:36.282 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.281+0000 7f702effd700 1 -- 172.21.15.58:0/2264825522 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7030048b20 con 0x7f70400f5580 2021-11-20T05:36:36.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:36 smithi058 conmon[33537]: cluster 2021-11-20T05:36:35.338931+0000 mon.smithi058 (mon.0) 666 : cluster [DBG] mds.? [v2:172.21.15.58:6826/444955435,v1:172.21.15.58:6827/444955435] up:rejoin 2021-11-20T05:36:36.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:36 smithi058 conmon[33537]: cluster 2021-11-20T05:36:35.338992+0000 mon.smithi058 (mon.0) 667 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:36.476 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.478+0000 7f7044d4b700 1 -- 172.21.15.58:0/2264825522 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7040027a00 con 0x7f70400f5580 2021-11-20T05:36:36.476 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.479+0000 7f702effd700 1 -- 172.21.15.58:0/2264825522 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2944 (secure 0 0 0) 0x7f703004c140 con 0x7f70400f5580 2021-11-20T05:36:36.477 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:36.477 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":45},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:35.335786+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14572,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14572":{"gid":14572,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":53,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/444955435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":444955435},{"type":"v1","addr":"172.21.15.58:6827","nonce":444955435}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:36.479 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.480+0000 7f7044d4b700 1 -- 172.21.15.58:0/2264825522 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f702805c2a0 msgr2=0x7f702805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:36.479 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.480+0000 7f7044d4b700 1 --2- 172.21.15.58:0/2264825522 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f702805c2a0 0x7f702805e760 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f7034013580 tx=0x7f7034009b30).stop 2021-11-20T05:36:36.479 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.480+0000 7f7044d4b700 1 -- 172.21.15.58:0/2264825522 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f5580 msgr2=0x7f70400604d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:36.480 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.480+0000 7f7044d4b700 1 --2- 172.21.15.58:0/2264825522 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f5580 0x7f70400604d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f70300093f0 tx=0x7f7030006ec0).stop 2021-11-20T05:36:36.480 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.481+0000 7f7044d4b700 1 -- 172.21.15.58:0/2264825522 shutdown_connections 2021-11-20T05:36:36.480 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.481+0000 7f7044d4b700 1 --2- 172.21.15.58:0/2264825522 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f702805c2a0 0x7f702805e760 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:36.480 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.481+0000 7f7044d4b700 1 --2- 172.21.15.58:0/2264825522 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f5580 0x7f70400604d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:36.481 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.481+0000 7f7044d4b700 1 --2- 172.21.15.58:0/2264825522 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70400f6820 0x7f704010e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:36.481 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.481+0000 7f7044d4b700 1 -- 172.21.15.58:0/2264825522 >> 172.21.15.58:0/2264825522 conn(0x7f70400f0ce0 msgr2=0x7f70400fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:36.481 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.481+0000 7f7044d4b700 1 -- 172.21.15.58:0/2264825522 shutdown_connections 2021-11-20T05:36:36.481 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:36.481+0000 7f7044d4b700 1 -- 172.21.15.58:0/2264825522 wait complete. 2021-11-20T05:36:36.481 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 57 2021-11-20T05:36:36.500 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:35.335786+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14572, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14572': {'gid': 14572, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 53, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/444955435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 444955435}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 444955435}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:36.501 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14572:up:rejoin 2021-11-20T05:36:36.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:36 smithi187 conmon[36358]: cluster 2021-11-20T05:36:35.338931+0000 mon.smithi058 (mon.0) 666 : cluster [DBG] mds.? [v2:172.21.15.58:6826/444955435,v1:172.21.15.58:6827/444955435] up:rejoin 2021-11-20T05:36:36.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:36 smithi187 conmon[36358]: cluster 2021-11-20T05:36:35.338992+0000 mon.smithi058 (mon.0) 667 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:37.502 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:37.638 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.638+0000 7f1069b55700 1 -- 172.21.15.58:0/1436286659 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10640f9760 msgr2=0x7f10640fbbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:37.638 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.638+0000 7f1069b55700 1 --2- 172.21.15.58:0/1436286659 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10640f9760 0x7f10640fbbb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f1050004660 tx=0x7f1050009b30).stop 2021-11-20T05:36:37.639 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.638+0000 7f1069b55700 1 -- 172.21.15.58:0/1436286659 shutdown_connections 2021-11-20T05:36:37.639 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.638+0000 7f1069b55700 1 --2- 172.21.15.58:0/1436286659 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10640f9760 0x7f10640fbbb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:37.639 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.638+0000 7f1069b55700 1 --2- 172.21.15.58:0/1436286659 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1064060450 0x7f10640f91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:36:37.640 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.638+0000 7f1069b55700 1 -- 172.21.15.58:0/1436286659 >> 172.21.15.58:0/1436286659 conn(0x7f10640f0c40 msgr2=0x7f10640f3080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:37.640 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.638+0000 7f1069b55700 1 -- 172.21.15.58:0/1436286659 shutdown_connections 2021-11-20T05:36:37.640 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.639+0000 7f1069b55700 1 -- 172.21.15.58:0/1436286659 wait complete. 2021-11-20T05:36:37.640 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.639+0000 7f1069b55700 1 Processor -- start 2021-11-20T05:36:37.640 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.639+0000 7f1069b55700 1 -- start start 2021-11-20T05:36:37.641 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.639+0000 7f1069b55700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1064060450 0x7f1064106dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:37.641 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.639+0000 7f1069b55700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10640f9760 0x7f10641072c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:37.641 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.639+0000 7f1069b55700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1064107950 con 0x7f10640f9760 2021-11-20T05:36:37.641 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.639+0000 7f1069b55700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1064108100 con 0x7f1064060450 2021-11-20T05:36:37.642 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.640+0000 7f1063fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10640f9760 0x7f10641072c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:37.642 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.640+0000 7f1063fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10640f9760 0x7f10641072c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41202/0 (socket says 172.21.15.58:41202) 2021-11-20T05:36:37.642 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.640+0000 7f1063fff700 1 -- 172.21.15.58:0/963974417 learned_addr learned my addr 172.21.15.58:0/963974417 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:37.642 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.640+0000 7f1068b53700 1 --2- 172.21.15.58:0/963974417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1064060450 0x7f1064106dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:37.643 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.640+0000 7f1063fff700 1 -- 172.21.15.58:0/963974417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1064060450 msgr2=0x7f1064106dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:37.643 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.640+0000 7f1063fff700 1 --2- 172.21.15.58:0/963974417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1064060450 0x7f1064106dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:37.643 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.640+0000 7f1063fff700 1 -- 172.21.15.58:0/963974417 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1050005040 con 0x7f10640f9760 2021-11-20T05:36:37.643 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.640+0000 7f1063fff700 1 --2- 172.21.15.58:0/963974417 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10640f9760 0x7f10641072c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f1050009f50 tx=0x7f1050009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:37.644 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.640+0000 7f10617fa700 1 -- 172.21.15.58:0/963974417 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1050018070 con 0x7f10640f9760 2021-11-20T05:36:37.644 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.640+0000 7f10617fa700 1 -- 172.21.15.58:0/963974417 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1050013410 con 0x7f10640f9760 2021-11-20T05:36:37.644 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.640+0000 7f1069b55700 1 -- 172.21.15.58:0/963974417 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1064108380 con 0x7f10640f9760 2021-11-20T05:36:37.645 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.640+0000 7f10617fa700 1 -- 172.21.15.58:0/963974417 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f105001c620 con 0x7f10640f9760 2021-11-20T05:36:37.645 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.640+0000 7f1069b55700 1 -- 172.21.15.58:0/963974417 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10641088e0 con 0x7f10640f9760 2021-11-20T05:36:37.646 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.641+0000 7f1069b55700 1 -- 172.21.15.58:0/963974417 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10640f48f0 con 0x7f10640f9760 2021-11-20T05:36:37.646 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.641+0000 7f10617fa700 1 -- 172.21.15.58:0/963974417 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f105001d420 con 0x7f10640f9760 2021-11-20T05:36:37.647 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.642+0000 7f10617fa700 1 --2- 172.21.15.58:0/963974417 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f105405c300 0x7f105405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:37.647 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.642+0000 7f10617fa700 1 -- 172.21.15.58:0/963974417 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f1050078040 con 0x7f10640f9760 2021-11-20T05:36:37.647 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.642+0000 7f1068b53700 1 --2- 172.21.15.58:0/963974417 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f105405c300 0x7f105405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:37.647 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.642+0000 7f1068b53700 1 --2- 172.21.15.58:0/963974417 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f105405c300 0x7f105405e7c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f1064107b90 tx=0x7f1058008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:37.648 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.646+0000 7f10617fa700 1 -- 172.21.15.58:0/963974417 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f105004aa50 con 0x7f10640f9760 2021-11-20T05:36:37.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:37 smithi187 conmon[36358]: cluster 2021-11-20T05:36:36.261736+0000 mgr.smithi058.zagdlk (mgr.14164) 267 : cluster [DBG] pgmap v237: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:36:37.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:37 smithi187 conmon[36358]: cluster 2021-11-20T05:36:36.346352+0000 mon.smithi058 (mon.0) 668 : cluster [DBG] mds.? [v2:172.21.15.187:6826/2249219455,v1:172.21.15.187:6827/2249219455] up:standby 2021-11-20T05:36:37.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:37 smithi187 conmon[36358]: cluster 2021-11-20 2021-11-20T05:36:37.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:37 smithi187 conmon[36358]: T05:36:36.346418+0000 mon.smithi058 (mon 2021-11-20T05:36:37.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:37 smithi187 conmon[36358]: .0) 669 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:37.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:37 smithi187 conmon[36358]: audit 2021 2021-11-20T05:36:37.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:37 smithi187 conmon[36358]: -11-20T05:36:36. 2021-11-20T05:36:37.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:37 smithi187 conmon[36358]: 479824+0000 mon.smithi058 (mon.0) 670 : audit [DBG] 2021-11-20T05:36:37.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:37 smithi187 conmon[36358]: from='client.? 172.21.15.58:0/2264825522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:37.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:37 smithi058 conmon[33537]: cluster 2021-11-20T05:36:36.261736+0000 mgr.smithi058.zagdlk (mgr.14164) 267 : 2021-11-20T05:36:37.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:37 smithi058 conmon[33537]: cluster [DBG] pgmap v237: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:36:37.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:37 smithi058 conmon[33537]: cluster 2021-11-20T05:36:36.346352+0000 mon.smithi058 (mon.0) 668 : cluster [DBG] mds.? [v2:172.21.15.187:6826/2249219455,v1:172.21.15.187:6827/2249219455] up:standby 2021-11-20T05:36:37.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:37 smithi058 conmon[33537]: cluster 2021-11-20T05:36:36.346418+0000 mon.smithi058 (mon.0) 669 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:37.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:37 smithi058 conmon[33537]: audit 2021-11-20T05:36:36.479824+0000 mon.smithi058 (mon.0) 670 : audit [DBG] from='client.? 172.21.15.58:0/2264825522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:37.839 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.842+0000 7f1069b55700 1 -- 172.21.15.58:0/963974417 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10640feda0 con 0x7f10640f9760 2021-11-20T05:36:37.840 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.842+0000 7f10617fa700 1 -- 172.21.15.58:0/963974417 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2944 (secure 0 0 0) 0x7f105004e070 con 0x7f10640f9760 2021-11-20T05:36:37.840 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:37.841 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":45},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:35.335786+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14572,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14572":{"gid":14572,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":53,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/444955435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":444955435},{"type":"v1","addr":"172.21.15.58:6827","nonce":444955435}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:37.842 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.844+0000 7f1069b55700 1 -- 172.21.15.58:0/963974417 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f105405c300 msgr2=0x7f105405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:37.842 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.844+0000 7f1069b55700 1 --2- 172.21.15.58:0/963974417 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f105405c300 0x7f105405e7c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f1064107b90 tx=0x7f1058008040).stop 2021-11-20T05:36:37.842 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.844+0000 7f1069b55700 1 -- 172.21.15.58:0/963974417 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10640f9760 msgr2=0x7f10641072c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:37.843 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.844+0000 7f1069b55700 1 --2- 172.21.15.58:0/963974417 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10640f9760 0x7f10641072c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f1050009f50 tx=0x7f1050009b30).stop 2021-11-20T05:36:37.843 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.844+0000 7f1069b55700 1 -- 172.21.15.58:0/963974417 shutdown_connections 2021-11-20T05:36:37.843 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.844+0000 7f1069b55700 1 --2- 172.21.15.58:0/963974417 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f105405c300 0x7f105405e7c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:37.843 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.844+0000 7f1069b55700 1 --2- 172.21.15.58:0/963974417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1064060450 0x7f1064106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:37.844 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.844+0000 7f1069b55700 1 --2- 172.21.15.58:0/963974417 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10640f9760 0x7f10641072c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:37.844 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.844+0000 7f1069b55700 1 -- 172.21.15.58:0/963974417 >> 172.21.15.58:0/963974417 conn(0x7f10640f0c40 msgr2=0x7f10640f3080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:37.844 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.844+0000 7f1069b55700 1 -- 172.21.15.58:0/963974417 shutdown_connections 2021-11-20T05:36:37.844 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:37.844+0000 7f1069b55700 1 -- 172.21.15.58:0/963974417 wait complete. 2021-11-20T05:36:37.844 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 57 2021-11-20T05:36:37.853 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:35.335786+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14572, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14572': {'gid': 14572, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 53, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/444955435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 444955435}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 444955435}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:37.854 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14572:up:rejoin 2021-11-20T05:36:38.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:38 smithi187 conmon[36358]: audit 2021-11-20T05:36:37.843223+0000 mon.smithi058 (mon.0) 671 : audit 2021-11-20T05:36:38.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:38 smithi187 conmon[36358]: [DBG] from='client.? 172.21.15.58:0/963974417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:38.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:38 smithi058 conmon[33537]: audit 2021-11-20T05:36:37.843223+0000 2021-11-20T05:36:38.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:38 smithi058 conmon[33537]: mon.smithi058 (mon.0) 671 : audit [DBG] from='client.? 172.21.15.58:0/963974417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:38.855 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:38.990 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.991+0000 7f69ec6d7700 1 -- 172.21.15.58:0/2283392792 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f69e40f55a0 msgr2=0x7f69e40f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:38.991 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.991+0000 7f69ec6d7700 1 --2- 172.21.15.58:0/2283392792 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f69e40f55a0 0x7f69e40f59c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f69d4004660 tx=0x7f69d4009b30).stop 2021-11-20T05:36:38.991 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.991+0000 7f69ec6d7700 1 -- 172.21.15.58:0/2283392792 shutdown_connections 2021-11-20T05:36:38.991 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.991+0000 7f69ec6d7700 1 --2- 172.21.15.58:0/2283392792 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69e40f6840 0x7f69e40f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:38.992 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.991+0000 7f69ec6d7700 1 --2- 172.21.15.58:0/2283392792 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f69e40f55a0 0x7f69e40f59c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:38.992 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.991+0000 7f69ec6d7700 1 -- 172.21.15.58:0/2283392792 >> 172.21.15.58:0/2283392792 conn(0x7f69e40f0ce0 msgr2=0x7f69e40f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:38.992 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.991+0000 7f69ec6d7700 1 -- 172.21.15.58:0/2283392792 shutdown_connections 2021-11-20T05:36:38.992 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.991+0000 7f69ec6d7700 1 -- 172.21.15.58:0/2283392792 wait complete. 2021-11-20T05:36:38.992 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.992+0000 7f69ec6d7700 1 Processor -- start 2021-11-20T05:36:38.993 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.992+0000 7f69ec6d7700 1 -- start start 2021-11-20T05:36:38.993 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.992+0000 7f69ec6d7700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f69e40f55a0 0x7f69e4106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:38.993 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.992+0000 7f69ec6d7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69e40f6840 0x7f69e41073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:38.993 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.992+0000 7f69ec6d7700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e4107960 con 0x7f69e40f55a0 2021-11-20T05:36:38.994 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.992+0000 7f69ec6d7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e4107aa0 con 0x7f69e40f6840 2021-11-20T05:36:38.994 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.992+0000 7f69ea473700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f69e40f55a0 0x7f69e4106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:38.994 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.992+0000 7f69ea473700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f69e40f55a0 0x7f69e4106ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41214/0 (socket says 172.21.15.58:41214) 2021-11-20T05:36:38.994 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.992+0000 7f69ea473700 1 -- 172.21.15.58:0/4276347054 learned_addr learned my addr 172.21.15.58:0/4276347054 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:38.995 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.992+0000 7f69e9c72700 1 --2- 172.21.15.58:0/4276347054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69e40f6840 0x7f69e41073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:38.995 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.992+0000 7f69ea473700 1 -- 172.21.15.58:0/4276347054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69e40f6840 msgr2=0x7f69e41073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:38.995 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.992+0000 7f69ea473700 1 --2- 172.21.15.58:0/4276347054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69e40f6840 0x7f69e41073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:38.995 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.992+0000 7f69ea473700 1 -- 172.21.15.58:0/4276347054 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d4005040 con 0x7f69e40f55a0 2021-11-20T05:36:38.995 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.992+0000 7f69e9c72700 1 --2- 172.21.15.58:0/4276347054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69e40f6840 0x7f69e41073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:36:38.996 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.992+0000 7f69ea473700 1 --2- 172.21.15.58:0/4276347054 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f69e40f55a0 0x7f69e4106ec0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f69d4004e30 tx=0x7f69d4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:38.996 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.993+0000 7f69daffd700 1 -- 172.21.15.58:0/4276347054 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69d4018070 con 0x7f69e40f55a0 2021-11-20T05:36:38.996 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.993+0000 7f69daffd700 1 -- 172.21.15.58:0/4276347054 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69d4013410 con 0x7f69e40f55a0 2021-11-20T05:36:38.996 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.993+0000 7f69ec6d7700 1 -- 172.21.15.58:0/4276347054 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e410c4e0 con 0x7f69e40f55a0 2021-11-20T05:36:38.997 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.993+0000 7f69daffd700 1 -- 172.21.15.58:0/4276347054 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69d401c3c0 con 0x7f69e40f55a0 2021-11-20T05:36:38.997 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.993+0000 7f69ec6d7700 1 -- 172.21.15.58:0/4276347054 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e410caa0 con 0x7f69e40f55a0 2021-11-20T05:36:38.997 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.993+0000 7f69ec6d7700 1 -- 172.21.15.58:0/4276347054 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e40fdd30 con 0x7f69e40f55a0 2021-11-20T05:36:38.998 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.994+0000 7f69daffd700 1 -- 172.21.15.58:0/4276347054 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f69d401d420 con 0x7f69e40f55a0 2021-11-20T05:36:38.998 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.994+0000 7f69daffd700 1 --2- 172.21.15.58:0/4276347054 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f69d005c2a0 0x7f69d005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:38.999 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.994+0000 7f69daffd700 1 -- 172.21.15.58:0/4276347054 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f69d4077e10 con 0x7f69e40f55a0 2021-11-20T05:36:38.999 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.994+0000 7f69e9c72700 1 --2- 172.21.15.58:0/4276347054 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f69d005c2a0 0x7f69d005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:38.999 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.995+0000 7f69e9c72700 1 --2- 172.21.15.58:0/4276347054 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f69d005c2a0 0x7f69d005e760 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f69e0009740 tx=0x7f69e0009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:38.999 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:38.998+0000 7f69daffd700 1 -- 172.21.15.58:0/4276347054 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69d404a820 con 0x7f69e40f55a0 2021-11-20T05:36:39.193 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:39.195+0000 7f69ec6d7700 1 -- 172.21.15.58:0/4276347054 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69e410d1a0 con 0x7f69e40f55a0 2021-11-20T05:36:39.193 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:39.196+0000 7f69daffd700 1 -- 172.21.15.58:0/4276347054 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2944 (secure 0 0 0) 0x7f69d404de40 con 0x7f69e40f55a0 2021-11-20T05:36:39.194 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:39.194 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":45},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:35.335786+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14572,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14572":{"gid":14572,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":53,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/444955435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":444955435},{"type":"v1","addr":"172.21.15.58:6827","nonce":444955435}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:39.196 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:39.197+0000 7f69ec6d7700 1 -- 172.21.15.58:0/4276347054 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f69d005c2a0 msgr2=0x7f69d005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:39.196 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:39.197+0000 7f69ec6d7700 1 --2- 172.21.15.58:0/4276347054 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f69d005c2a0 0x7f69d005e760 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f69e0009740 tx=0x7f69e0009be0).stop 2021-11-20T05:36:39.196 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:39.197+0000 7f69ec6d7700 1 -- 172.21.15.58:0/4276347054 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f69e40f55a0 msgr2=0x7f69e4106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:39.196 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:39.197+0000 7f69ec6d7700 1 --2- 172.21.15.58:0/4276347054 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f69e40f55a0 0x7f69e4106ec0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f69d4004e30 tx=0x7f69d4009b30).stop 2021-11-20T05:36:39.197 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:39.197+0000 7f69ec6d7700 1 -- 172.21.15.58:0/4276347054 shutdown_connections 2021-11-20T05:36:39.197 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:39.197+0000 7f69ec6d7700 1 --2- 172.21.15.58:0/4276347054 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f69d005c2a0 0x7f69d005e760 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:39.197 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:39.197+0000 7f69ec6d7700 1 --2- 172.21.15.58:0/4276347054 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f69e40f55a0 0x7f69e4106ec0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:39.197 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:39.197+0000 7f69ec6d7700 1 --2- 172.21.15.58:0/4276347054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69e40f6840 0x7f69e41073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:39.198 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:39.197+0000 7f69ec6d7700 1 -- 172.21.15.58:0/4276347054 >> 172.21.15.58:0/4276347054 conn(0x7f69e40f0ce0 msgr2=0x7f69e405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:39.198 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:39.198+0000 7f69ec6d7700 1 -- 172.21.15.58:0/4276347054 shutdown_connections 2021-11-20T05:36:39.198 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:39.198+0000 7f69ec6d7700 1 -- 172.21.15.58:0/4276347054 wait complete. 2021-11-20T05:36:39.198 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 57 2021-11-20T05:36:39.207 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:35.335786+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14572, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14572': {'gid': 14572, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 53, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/444955435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 444955435}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 444955435}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:39.207 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14572:up:rejoin 2021-11-20T05:36:39.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:39 smithi058 conmon[33537]: cluster 2021-11-20T05:36:38.262217+0000 mgr.smithi058.zagdlk (mgr.14164) 268 : cluster [DBG] pgmap v238: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:39.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:39 smithi058 conmon[33537]: audit 2021-11-20T05:36:39.196811+0000 mon.smithi058 (mon.0) 672 : audit [DBG] from='client.? 172.21.15.58:0/4276347054' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:39.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:39 smithi187 conmon[36358]: cluster 2021-11-20T05:36:38.262217+0000 mgr.smithi058.zagdlk (mgr.14164) 268 : cluster [DBG] pgmap v238: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:39.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:39 smithi187 conmon[36358]: audit 2021-11-20T05:36:39.196811+0000 mon.smithi058 (mon.0) 672 : audit [DBG] from='client.? 172.21.15.58:0/4276347054' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:40.209 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:40.352 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.352+0000 7f2d1d493700 1 -- 172.21.15.58:0/921758547 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2d180f6820 msgr2=0x7f2d180f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:40.352 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.352+0000 7f2d1d493700 1 --2- 172.21.15.58:0/921758547 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2d180f6820 0x7f2d180f6c80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f2d0c004660 tx=0x7f2d0c009b30).stop 2021-11-20T05:36:40.353 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.352+0000 7f2d1d493700 1 -- 172.21.15.58:0/921758547 shutdown_connections 2021-11-20T05:36:40.353 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.352+0000 7f2d1d493700 1 --2- 172.21.15.58:0/921758547 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2d180f6820 0x7f2d180f6c80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:40.353 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.352+0000 7f2d1d493700 1 --2- 172.21.15.58:0/921758547 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d180f5580 0x7f2d180f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:40.353 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.352+0000 7f2d1d493700 1 -- 172.21.15.58:0/921758547 >> 172.21.15.58:0/921758547 conn(0x7f2d180f0ce0 msgr2=0x7f2d180f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:40.354 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.353+0000 7f2d1d493700 1 -- 172.21.15.58:0/921758547 shutdown_connections 2021-11-20T05:36:40.354 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.353+0000 7f2d1d493700 1 -- 172.21.15.58:0/921758547 wait complete. 2021-11-20T05:36:40.354 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.353+0000 7f2d1d493700 1 Processor -- start 2021-11-20T05:36:40.354 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.353+0000 7f2d1d493700 1 -- start start 2021-11-20T05:36:40.355 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.353+0000 7f2d1d493700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d180f5580 0x7f2d18105dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:40.355 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.353+0000 7f2d1d493700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2d180f6820 0x7f2d181062b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:40.355 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.353+0000 7f2d1d493700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d18106860 con 0x7f2d180f6820 2021-11-20T05:36:40.355 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.353+0000 7f2d1d493700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d181069a0 con 0x7f2d180f5580 2021-11-20T05:36:40.356 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.354+0000 7f2d167fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2d180f6820 0x7f2d181062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:40.356 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.354+0000 7f2d167fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2d180f6820 0x7f2d181062b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41228/0 (socket says 172.21.15.58:41228) 2021-11-20T05:36:40.356 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.354+0000 7f2d167fc700 1 -- 172.21.15.58:0/2820690071 learned_addr learned my addr 172.21.15.58:0/2820690071 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:40.356 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.354+0000 7f2d16ffd700 1 --2- 172.21.15.58:0/2820690071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d180f5580 0x7f2d18105dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:40.357 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.354+0000 7f2d167fc700 1 -- 172.21.15.58:0/2820690071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d180f5580 msgr2=0x7f2d18105dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:40.357 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.354+0000 7f2d167fc700 1 --2- 172.21.15.58:0/2820690071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d180f5580 0x7f2d18105dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:40.357 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.354+0000 7f2d167fc700 1 -- 172.21.15.58:0/2820690071 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d0c005040 con 0x7f2d180f6820 2021-11-20T05:36:40.357 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.354+0000 7f2d16ffd700 1 --2- 172.21.15.58:0/2820690071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d180f5580 0x7f2d18105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:36:40.357 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.354+0000 7f2d167fc700 1 --2- 172.21.15.58:0/2820690071 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2d180f6820 0x7f2d181062b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f2d0c009830 tx=0x7f2d0c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:40.358 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.354+0000 7f2cff7fe700 1 -- 172.21.15.58:0/2820690071 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d0c018070 con 0x7f2d180f6820 2021-11-20T05:36:40.358 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.354+0000 7f2cff7fe700 1 -- 172.21.15.58:0/2820690071 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d0c013410 con 0x7f2d180f6820 2021-11-20T05:36:40.358 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.355+0000 7f2d1d493700 1 -- 172.21.15.58:0/2820690071 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1810b3e0 con 0x7f2d180f6820 2021-11-20T05:36:40.358 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.355+0000 7f2cff7fe700 1 -- 172.21.15.58:0/2820690071 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d0c01c620 con 0x7f2d180f6820 2021-11-20T05:36:40.359 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.355+0000 7f2d1d493700 1 -- 172.21.15.58:0/2820690071 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1810b9a0 con 0x7f2d180f6820 2021-11-20T05:36:40.360 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.355+0000 7f2d1d493700 1 -- 172.21.15.58:0/2820690071 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d18027a00 con 0x7f2d180f6820 2021-11-20T05:36:40.360 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.356+0000 7f2cff7fe700 1 -- 172.21.15.58:0/2820690071 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2d0c01d420 con 0x7f2d180f6820 2021-11-20T05:36:40.360 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.356+0000 7f2cff7fe700 1 --2- 172.21.15.58:0/2820690071 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2d0005c2a0 0x7f2d0005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:40.361 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.356+0000 7f2cff7fe700 1 -- 172.21.15.58:0/2820690071 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f2d0c077f90 con 0x7f2d180f6820 2021-11-20T05:36:40.361 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.356+0000 7f2d16ffd700 1 --2- 172.21.15.58:0/2820690071 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2d0005c2a0 0x7f2d0005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:40.361 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.357+0000 7f2d16ffd700 1 --2- 172.21.15.58:0/2820690071 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2d0005c2a0 0x7f2d0005e760 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f2d180f6680 tx=0x7f2d08009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:40.361 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.360+0000 7f2cff7fe700 1 -- 172.21.15.58:0/2820690071 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d0c04aa20 con 0x7f2d180f6820 2021-11-20T05:36:40.556 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.559+0000 7f2d1d493700 1 -- 172.21.15.58:0/2820690071 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d180fdd10 con 0x7f2d180f6820 2021-11-20T05:36:40.557 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.559+0000 7f2cff7fe700 1 -- 172.21.15.58:0/2820690071 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2944 (secure 0 0 0) 0x7f2d0c04e040 con 0x7f2d180f6820 2021-11-20T05:36:40.557 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:40.558 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":45},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:35.335786+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14572,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14572":{"gid":14572,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":53,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/444955435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":444955435},{"type":"v1","addr":"172.21.15.58:6827","nonce":444955435}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:40.559 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.561+0000 7f2d1d493700 1 -- 172.21.15.58:0/2820690071 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2d0005c2a0 msgr2=0x7f2d0005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:40.559 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.561+0000 7f2d1d493700 1 --2- 172.21.15.58:0/2820690071 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2d0005c2a0 0x7f2d0005e760 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f2d180f6680 tx=0x7f2d08009040).stop 2021-11-20T05:36:40.560 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.561+0000 7f2d1d493700 1 -- 172.21.15.58:0/2820690071 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2d180f6820 msgr2=0x7f2d181062b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:40.560 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.561+0000 7f2d1d493700 1 --2- 172.21.15.58:0/2820690071 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2d180f6820 0x7f2d181062b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f2d0c009830 tx=0x7f2d0c009b30).stop 2021-11-20T05:36:40.560 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.561+0000 7f2d1d493700 1 -- 172.21.15.58:0/2820690071 shutdown_connections 2021-11-20T05:36:40.560 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.561+0000 7f2d1d493700 1 --2- 172.21.15.58:0/2820690071 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2d0005c2a0 0x7f2d0005e760 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:40.561 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.561+0000 7f2d1d493700 1 --2- 172.21.15.58:0/2820690071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d180f5580 0x7f2d18105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:40.561 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.561+0000 7f2d1d493700 1 --2- 172.21.15.58:0/2820690071 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2d180f6820 0x7f2d181062b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:40.561 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.561+0000 7f2d1d493700 1 -- 172.21.15.58:0/2820690071 >> 172.21.15.58:0/2820690071 conn(0x7f2d180f0ce0 msgr2=0x7f2d180fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:40.561 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.561+0000 7f2d1d493700 1 -- 172.21.15.58:0/2820690071 shutdown_connections 2021-11-20T05:36:40.561 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:40.561+0000 7f2d1d493700 1 -- 172.21.15.58:0/2820690071 wait complete. 2021-11-20T05:36:40.562 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 57 2021-11-20T05:36:40.571 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:35.335786+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14572, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14572': {'gid': 14572, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 53, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/444955435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 444955435}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 444955435}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:40.571 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14572:up:rejoin 2021-11-20T05:36:41.572 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:41.602 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:41 smithi058 conmon[33537]: cluster 2021-11-20T05:36:40.267352+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:36:41.603 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:41 smithi058 conmon[33537]: 269 : cluster [DBG] pgmap v239: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:41.603 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:41 smithi058 conmon[33537]: audit 2021-11-20T05:36:40.560327+0000 mon.smithi058 (mon.0) 673 : audit 2021-11-20T05:36:41.603 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:41 smithi058 conmon[33537]: [DBG] from='client.? 172.21.15.58:0/2820690071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:41.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:41 smithi187 conmon[36358]: cluster 2021-11-20T05:36:40.267352+0000 mgr.smithi058.zagdlk (mgr.14164) 269 : cluster [DBG] pgmap v239: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:41.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:41 smithi187 conmon[36358]: audit 2021-11-20T05:36:40.560327+0000 mon.smithi058 (mon.0) 673 : audit [DBG] from='client.? 172.21.15.58:0/2820690071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:41.706 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.708+0000 7fec1ad7a700 1 -- 172.21.15.58:0/173515877 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec140f4a00 msgr2=0x7fec140f4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:41.706 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.708+0000 7fec1ad7a700 1 --2- 172.21.15.58:0/173515877 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec140f4a00 0x7fec140f4e60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fec00004770 tx=0x7fec0000ac40).stop 2021-11-20T05:36:41.707 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.708+0000 7fec1ad7a700 1 -- 172.21.15.58:0/173515877 shutdown_connections 2021-11-20T05:36:41.707 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.708+0000 7fec1ad7a700 1 --2- 172.21.15.58:0/173515877 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec140f4a00 0x7fec140f4e60 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:41.707 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.708+0000 7fec1ad7a700 1 --2- 172.21.15.58:0/173515877 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec140f63f0 0x7fec140f6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:41.707 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.708+0000 7fec1ad7a700 1 -- 172.21.15.58:0/173515877 >> 172.21.15.58:0/173515877 conn(0x7fec140f04a0 msgr2=0x7fec140f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:41.708 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.708+0000 7fec1ad7a700 1 -- 172.21.15.58:0/173515877 shutdown_connections 2021-11-20T05:36:41.708 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.708+0000 7fec1ad7a700 1 -- 172.21.15.58:0/173515877 wait complete. 2021-11-20T05:36:41.708 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.709+0000 7fec1ad7a700 1 Processor -- start 2021-11-20T05:36:41.708 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.709+0000 7fec1ad7a700 1 -- start start 2021-11-20T05:36:41.711 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.709+0000 7fec1ad7a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec140f4a00 0x7fec14106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:41.712 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.709+0000 7fec1ad7a700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec140f63f0 0x7fec141073e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:41.712 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.709+0000 7fec1ad7a700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec14107990 con 0x7fec140f63f0 2021-11-20T05:36:41.712 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.709+0000 7fec1ad7a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec14107ad0 con 0x7fec140f4a00 2021-11-20T05:36:41.712 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.710+0000 7fec13fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec140f63f0 0x7fec141073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:41.713 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.710+0000 7fec13fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec140f63f0 0x7fec141073e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41240/0 (socket says 172.21.15.58:41240) 2021-11-20T05:36:41.713 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.710+0000 7fec13fff700 1 -- 172.21.15.58:0/1377685110 learned_addr learned my addr 172.21.15.58:0/1377685110 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:41.713 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.710+0000 7fec18b16700 1 --2- 172.21.15.58:0/1377685110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec140f4a00 0x7fec14106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:41.713 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.710+0000 7fec13fff700 1 -- 172.21.15.58:0/1377685110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec140f4a00 msgr2=0x7fec14106ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:41.713 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.710+0000 7fec13fff700 1 --2- 172.21.15.58:0/1377685110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec140f4a00 0x7fec14106ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:41.714 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.710+0000 7fec13fff700 1 -- 172.21.15.58:0/1377685110 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec00005040 con 0x7fec140f63f0 2021-11-20T05:36:41.714 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.710+0000 7fec13fff700 1 --2- 172.21.15.58:0/1377685110 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec140f63f0 0x7fec141073e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fec00004f40 tx=0x7fec0000ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:41.714 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.710+0000 7fec117fa700 1 -- 172.21.15.58:0/1377685110 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec0000f030 con 0x7fec140f63f0 2021-11-20T05:36:41.714 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.710+0000 7fec117fa700 1 -- 172.21.15.58:0/1377685110 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec00007c60 con 0x7fec140f63f0 2021-11-20T05:36:41.715 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.710+0000 7fec1ad7a700 1 -- 172.21.15.58:0/1377685110 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1410c510 con 0x7fec140f63f0 2021-11-20T05:36:41.715 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.710+0000 7fec117fa700 1 -- 172.21.15.58:0/1377685110 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec00019070 con 0x7fec140f63f0 2021-11-20T05:36:41.715 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.710+0000 7fec1ad7a700 1 -- 172.21.15.58:0/1377685110 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1410cad0 con 0x7fec140f63f0 2021-11-20T05:36:41.715 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.711+0000 7fec1ad7a700 1 -- 172.21.15.58:0/1377685110 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec14027a00 con 0x7fec140f63f0 2021-11-20T05:36:41.716 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.712+0000 7fec117fa700 1 -- 172.21.15.58:0/1377685110 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fec000077b0 con 0x7fec140f63f0 2021-11-20T05:36:41.716 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.712+0000 7fec117fa700 1 --2- 172.21.15.58:0/1377685110 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fec0405c2a0 0x7fec0405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:41.716 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.712+0000 7fec117fa700 1 -- 172.21.15.58:0/1377685110 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fec00077bb0 con 0x7fec140f63f0 2021-11-20T05:36:41.716 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.712+0000 7fec18b16700 1 --2- 172.21.15.58:0/1377685110 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fec0405c2a0 0x7fec0405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:41.717 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.712+0000 7fec18b16700 1 --2- 172.21.15.58:0/1377685110 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fec0405c2a0 0x7fec0405e760 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fec140f5e40 tx=0x7fec0800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:41.717 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.716+0000 7fec117fa700 1 -- 172.21.15.58:0/1377685110 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec0007e080 con 0x7fec140f63f0 2021-11-20T05:36:41.910 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.912+0000 7fec1ad7a700 1 -- 172.21.15.58:0/1377685110 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec140fcd50 con 0x7fec140f63f0 2021-11-20T05:36:41.912 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.913+0000 7fec117fa700 1 -- 172.21.15.58:0/1377685110 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2944 (secure 0 0 0) 0x7fec0004a5c0 con 0x7fec140f63f0 2021-11-20T05:36:41.912 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:41.913 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":45},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:35.335786+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14572,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14572":{"gid":14572,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":53,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/444955435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":444955435},{"type":"v1","addr":"172.21.15.58:6827","nonce":444955435}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:41.914 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.914+0000 7fec1ad7a700 1 -- 172.21.15.58:0/1377685110 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fec0405c2a0 msgr2=0x7fec0405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:41.914 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.914+0000 7fec1ad7a700 1 --2- 172.21.15.58:0/1377685110 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fec0405c2a0 0x7fec0405e760 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fec140f5e40 tx=0x7fec0800b040).stop 2021-11-20T05:36:41.915 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.914+0000 7fec1ad7a700 1 -- 172.21.15.58:0/1377685110 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec140f63f0 msgr2=0x7fec141073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:41.915 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.914+0000 7fec1ad7a700 1 --2- 172.21.15.58:0/1377685110 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec140f63f0 0x7fec141073e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fec00004f40 tx=0x7fec0000ac40).stop 2021-11-20T05:36:41.915 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.915+0000 7fec1ad7a700 1 -- 172.21.15.58:0/1377685110 shutdown_connections 2021-11-20T05:36:41.915 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.915+0000 7fec1ad7a700 1 --2- 172.21.15.58:0/1377685110 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fec0405c2a0 0x7fec0405e760 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:41.916 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.915+0000 7fec1ad7a700 1 --2- 172.21.15.58:0/1377685110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec140f4a00 0x7fec14106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:41.916 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.915+0000 7fec1ad7a700 1 --2- 172.21.15.58:0/1377685110 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec140f63f0 0x7fec141073e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:41.916 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.915+0000 7fec1ad7a700 1 -- 172.21.15.58:0/1377685110 >> 172.21.15.58:0/1377685110 conn(0x7fec140f04a0 msgr2=0x7fec140f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:41.916 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.915+0000 7fec1ad7a700 1 -- 172.21.15.58:0/1377685110 shutdown_connections 2021-11-20T05:36:41.917 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:41.915+0000 7fec1ad7a700 1 -- 172.21.15.58:0/1377685110 wait complete. 2021-11-20T05:36:41.917 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 57 2021-11-20T05:36:41.924 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:35.335786+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14572, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14572': {'gid': 14572, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 53, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/444955435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 444955435}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 444955435}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:41.925 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14572:up:rejoin 2021-11-20T05:36:42.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:42 smithi187 conmon[36358]: audit 2021-11-20T05:36:41.914009+0000 mon.smithi058 (mon.0) 674 : audit [DBG] from='client.? 172.21.15.58:0/1377685110' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:42.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:42 smithi058 conmon[33537]: audit 2021-11-20T05:36:41.914009+0000 mon.smithi058 (mon.0 2021-11-20T05:36:42.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:42 smithi058 conmon[33537]: ) 674 : audit [DBG] from='client.? 172.21.15.58:0/1377685110' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:42.925 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:43.060 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.061+0000 7fcab29e6700 1 -- 172.21.15.58:0/459850836 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcaac0f6680 msgr2=0x7fcaac0f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:43.061 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.061+0000 7fcab29e6700 1 --2- 172.21.15.58:0/459850836 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcaac0f6680 0x7fcaac0f6aa0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fca9c004660 tx=0x7fca9c009b30).stop 2021-11-20T05:36:43.061 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.061+0000 7fcab29e6700 1 -- 172.21.15.58:0/459850836 shutdown_connections 2021-11-20T05:36:43.061 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.061+0000 7fcab29e6700 1 --2- 172.21.15.58:0/459850836 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaac0f7920 0x7fcaac0f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:43.061 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.061+0000 7fcab29e6700 1 --2- 172.21.15.58:0/459850836 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcaac0f6680 0x7fcaac0f6aa0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:43.061 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.061+0000 7fcab29e6700 1 -- 172.21.15.58:0/459850836 >> 172.21.15.58:0/459850836 conn(0x7fcaac0f1de0 msgr2=0x7fcaac0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:43.062 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.061+0000 7fcab29e6700 1 -- 172.21.15.58:0/459850836 shutdown_connections 2021-11-20T05:36:43.062 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.061+0000 7fcab29e6700 1 -- 172.21.15.58:0/459850836 wait complete. 2021-11-20T05:36:43.062 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.061+0000 7fcab29e6700 1 Processor -- start 2021-11-20T05:36:43.062 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.062+0000 7fcab29e6700 1 -- start start 2021-11-20T05:36:43.063 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.062+0000 7fcab29e6700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcaac0f6680 0x7fcaac106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:43.063 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.062+0000 7fcab29e6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaac0f7920 0x7fcaac1073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:43.063 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.062+0000 7fcab29e6700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac107950 con 0x7fcaac0f6680 2021-11-20T05:36:43.063 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.062+0000 7fcab29e6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac107a90 con 0x7fcaac0f7920 2021-11-20T05:36:43.064 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.062+0000 7fcaabfff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcaac0f6680 0x7fcaac106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:43.064 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.062+0000 7fcaabfff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcaac0f6680 0x7fcaac106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41248/0 (socket says 172.21.15.58:41248) 2021-11-20T05:36:43.064 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.062+0000 7fcaabfff700 1 -- 172.21.15.58:0/685925127 learned_addr learned my addr 172.21.15.58:0/685925127 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:43.064 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.062+0000 7fcaab7fe700 1 --2- 172.21.15.58:0/685925127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaac0f7920 0x7fcaac1073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:43.064 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.062+0000 7fcaabfff700 1 -- 172.21.15.58:0/685925127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaac0f7920 msgr2=0x7fcaac1073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:43.065 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.062+0000 7fcaabfff700 1 --2- 172.21.15.58:0/685925127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaac0f7920 0x7fcaac1073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:43.065 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.062+0000 7fcaabfff700 1 -- 172.21.15.58:0/685925127 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca9c005040 con 0x7fcaac0f6680 2021-11-20T05:36:43.065 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.062+0000 7fcaabfff700 1 --2- 172.21.15.58:0/685925127 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcaac0f6680 0x7fcaac106eb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fca9c007a60 tx=0x7fca9c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:43.065 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.063+0000 7fcaa8ff9700 1 -- 172.21.15.58:0/685925127 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca9c018070 con 0x7fcaac0f6680 2021-11-20T05:36:43.066 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.063+0000 7fcaa8ff9700 1 -- 172.21.15.58:0/685925127 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca9c013410 con 0x7fcaac0f6680 2021-11-20T05:36:43.066 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.063+0000 7fcab29e6700 1 -- 172.21.15.58:0/685925127 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaac10c4d0 con 0x7fcaac0f6680 2021-11-20T05:36:43.066 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.063+0000 7fcaa8ff9700 1 -- 172.21.15.58:0/685925127 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca9c01c3c0 con 0x7fcaac0f6680 2021-11-20T05:36:43.066 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.063+0000 7fcab29e6700 1 -- 172.21.15.58:0/685925127 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaac10ca90 con 0x7fcaac0f6680 2021-11-20T05:36:43.068 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.064+0000 7fcab29e6700 1 -- 172.21.15.58:0/685925127 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaac027a00 con 0x7fcaac0f6680 2021-11-20T05:36:43.068 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.064+0000 7fcaa8ff9700 1 -- 172.21.15.58:0/685925127 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fca9c01d420 con 0x7fcaac0f6680 2021-11-20T05:36:43.068 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.064+0000 7fcaa8ff9700 1 --2- 172.21.15.58:0/685925127 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fca9405c2a0 0x7fca9405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:43.068 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.064+0000 7fcaa8ff9700 1 -- 172.21.15.58:0/685925127 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fca9c077f70 con 0x7fcaac0f6680 2021-11-20T05:36:43.069 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.064+0000 7fcaab7fe700 1 --2- 172.21.15.58:0/685925127 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fca9405c2a0 0x7fca9405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:43.069 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.065+0000 7fcaab7fe700 1 --2- 172.21.15.58:0/685925127 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fca9405c2a0 0x7fca9405e760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fcaa0004c40 tx=0x7fcaa000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:43.069 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.068+0000 7fcaa8ff9700 1 -- 172.21.15.58:0/685925127 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca9c0a42b0 con 0x7fcaac0f6680 2021-11-20T05:36:43.261 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.263+0000 7fcab29e6700 1 -- 172.21.15.58:0/685925127 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcaac0fee10 con 0x7fcaac0f6680 2021-11-20T05:36:43.261 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.264+0000 7fcaa8ff9700 1 -- 172.21.15.58:0/685925127 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2944 (secure 0 0 0) 0x7fca9c04a980 con 0x7fcaac0f6680 2021-11-20T05:36:43.262 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:43.262 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":45},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:35.335786+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14572,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14572":{"gid":14572,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":53,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/444955435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":444955435},{"type":"v1","addr":"172.21.15.58:6827","nonce":444955435}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:43.263 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.265+0000 7fcab29e6700 1 -- 172.21.15.58:0/685925127 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fca9405c2a0 msgr2=0x7fca9405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:43.264 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.265+0000 7fcab29e6700 1 --2- 172.21.15.58:0/685925127 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fca9405c2a0 0x7fca9405e760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fcaa0004c40 tx=0x7fcaa000b040).stop 2021-11-20T05:36:43.264 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.266+0000 7fcab29e6700 1 -- 172.21.15.58:0/685925127 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcaac0f6680 msgr2=0x7fcaac106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:43.264 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.266+0000 7fcab29e6700 1 --2- 172.21.15.58:0/685925127 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcaac0f6680 0x7fcaac106eb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fca9c007a60 tx=0x7fca9c009b30).stop 2021-11-20T05:36:43.264 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.266+0000 7fcab29e6700 1 -- 172.21.15.58:0/685925127 shutdown_connections 2021-11-20T05:36:43.265 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.266+0000 7fcab29e6700 1 --2- 172.21.15.58:0/685925127 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fca9405c2a0 0x7fca9405e760 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:43.265 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.266+0000 7fcab29e6700 1 --2- 172.21.15.58:0/685925127 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcaac0f6680 0x7fcaac106eb0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:43.265 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.266+0000 7fcab29e6700 1 --2- 172.21.15.58:0/685925127 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaac0f7920 0x7fcaac1073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:43.265 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.266+0000 7fcab29e6700 1 -- 172.21.15.58:0/685925127 >> 172.21.15.58:0/685925127 conn(0x7fcaac0f1de0 msgr2=0x7fcaac0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:43.266 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.266+0000 7fcab29e6700 1 -- 172.21.15.58:0/685925127 shutdown_connections 2021-11-20T05:36:43.266 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:43.266+0000 7fcab29e6700 1 -- 172.21.15.58:0/685925127 wait complete. 2021-11-20T05:36:43.266 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 57 2021-11-20T05:36:43.276 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:35.335786+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14572, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14572': {'gid': 14572, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 53, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/444955435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 444955435}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 444955435}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:43.276 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14572:up:rejoin 2021-11-20T05:36:43.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:43 smithi187 conmon[36358]: cluster 2021-11-20T05:36:42.267891+0000 mgr.smithi058.zagdlk (mgr.14164) 270 : cluster [DBG] pgmap v240: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T05:36:43.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:43 smithi187 conmon[36358]: audit 2021-11-20T05:36:43.265049+0000 mon.smithi058 (mon.0) 675 : audit [DBG] from='client.? 172.21.15.58:0/685925127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:43.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:43 smithi058 conmon[33537]: cluster 2021-11-20T05:36: 2021-11-20T05:36:43.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:43 smithi058 conmon[33537]: 42.267891+0000 mgr.smithi058.zagdlk (mgr.14164) 270 : cluster [DBG] pgmap v240: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T05:36:43.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:43 smithi058 conmon[33537]: audit 2021-11-20T05:36:43.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:43 smithi058 conmon[33537]: 2021-11-20T05:36:43 2021-11-20T05:36:43.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:43 smithi058 conmon[33537]: .265049+0000 mon.smithi058 (mon. 2021-11-20T05:36:43.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:43 smithi058 conmon[33537]: 0) 675 : audit [DBG] from='client.? 172.21.15.58:0/685925127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:44.277 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:44.409 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.409+0000 7f1fa96ac700 1 -- 172.21.15.58:0/3127909983 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1fa40f6820 msgr2=0x7f1fa40f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:44.410 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.409+0000 7f1fa96ac700 1 --2- 172.21.15.58:0/3127909983 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1fa40f6820 0x7f1fa40f6c80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f1f8c004660 tx=0x7f1f8c009b30).stop 2021-11-20T05:36:44.410 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.409+0000 7f1fa96ac700 1 -- 172.21.15.58:0/3127909983 shutdown_connections 2021-11-20T05:36:44.410 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.409+0000 7f1fa96ac700 1 --2- 172.21.15.58:0/3127909983 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1fa40f6820 0x7f1fa40f6c80 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:44.410 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.409+0000 7f1fa96ac700 1 --2- 172.21.15.58:0/3127909983 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1fa40f5580 0x7f1fa40f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:44.411 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.409+0000 7f1fa96ac700 1 -- 172.21.15.58:0/3127909983 >> 172.21.15.58:0/3127909983 conn(0x7f1fa40f0ce0 msgr2=0x7f1fa40f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:44.411 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.409+0000 7f1fa96ac700 1 -- 172.21.15.58:0/3127909983 shutdown_connections 2021-11-20T05:36:44.411 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.409+0000 7f1fa96ac700 1 -- 172.21.15.58:0/3127909983 wait complete. 2021-11-20T05:36:44.411 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.410+0000 7f1fa96ac700 1 Processor -- start 2021-11-20T05:36:44.412 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.410+0000 7f1fa96ac700 1 -- start start 2021-11-20T05:36:44.412 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.410+0000 7f1fa96ac700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1fa40f5580 0x7f1fa40f9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:44.412 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.410+0000 7f1fa96ac700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1fa40f6820 0x7f1fa40f7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:44.412 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.410+0000 7f1fa96ac700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa40f9ab0 con 0x7f1fa40f5580 2021-11-20T05:36:44.412 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.410+0000 7f1fa96ac700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa40f8050 con 0x7f1fa40f6820 2021-11-20T05:36:44.413 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.410+0000 7f1fa2ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1fa40f5580 0x7f1fa40f9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:44.413 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.410+0000 7f1fa2ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1fa40f5580 0x7f1fa40f9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41258/0 (socket says 172.21.15.58:41258) 2021-11-20T05:36:44.413 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.410+0000 7f1fa2ffd700 1 -- 172.21.15.58:0/2507608225 learned_addr learned my addr 172.21.15.58:0/2507608225 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:44.414 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.410+0000 7f1fa27fc700 1 --2- 172.21.15.58:0/2507608225 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1fa40f6820 0x7f1fa40f7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:44.414 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.411+0000 7f1fa2ffd700 1 -- 172.21.15.58:0/2507608225 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1fa40f6820 msgr2=0x7f1fa40f7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:44.414 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.411+0000 7f1fa2ffd700 1 --2- 172.21.15.58:0/2507608225 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1fa40f6820 0x7f1fa40f7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:44.414 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.411+0000 7f1fa2ffd700 1 -- 172.21.15.58:0/2507608225 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8c005040 con 0x7f1fa40f5580 2021-11-20T05:36:44.415 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.411+0000 7f1fa2ffd700 1 --2- 172.21.15.58:0/2507608225 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1fa40f5580 0x7f1fa40f9500 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f1f9400c530 tx=0x7f1f94007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:44.415 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.411+0000 7f1f9b7fe700 1 -- 172.21.15.58:0/2507608225 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f9400f650 con 0x7f1fa40f5580 2021-11-20T05:36:44.415 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.411+0000 7f1f9b7fe700 1 -- 172.21.15.58:0/2507608225 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f9400fc60 con 0x7f1fa40f5580 2021-11-20T05:36:44.415 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.411+0000 7f1fa96ac700 1 -- 172.21.15.58:0/2507608225 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa40f82d0 con 0x7f1fa40f5580 2021-11-20T05:36:44.416 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.411+0000 7f1f9b7fe700 1 -- 172.21.15.58:0/2507608225 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f94012c60 con 0x7f1fa40f5580 2021-11-20T05:36:44.416 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.411+0000 7f1fa96ac700 1 -- 172.21.15.58:0/2507608225 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa40f8890 con 0x7f1fa40f5580 2021-11-20T05:36:44.416 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.412+0000 7f1fa96ac700 1 -- 172.21.15.58:0/2507608225 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa40fdd10 con 0x7f1fa40f5580 2021-11-20T05:36:44.416 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.412+0000 7f1f9b7fe700 1 -- 172.21.15.58:0/2507608225 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1f94019030 con 0x7f1fa40f5580 2021-11-20T05:36:44.417 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.413+0000 7f1f9b7fe700 1 --2- 172.21.15.58:0/2507608225 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1f9005c2a0 0x7f1f9005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:44.417 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.413+0000 7f1f9b7fe700 1 -- 172.21.15.58:0/2507608225 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f1f94075d70 con 0x7f1fa40f5580 2021-11-20T05:36:44.418 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.413+0000 7f1fa27fc700 1 --2- 172.21.15.58:0/2507608225 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1f9005c2a0 0x7f1f9005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:44.418 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.413+0000 7f1fa27fc700 1 --2- 172.21.15.58:0/2507608225 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1f9005c2a0 0x7f1f9005e760 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f1f8c004630 tx=0x7f1f8c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:44.418 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.417+0000 7f1f9b7fe700 1 -- 172.21.15.58:0/2507608225 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f940462b0 con 0x7f1fa40f5580 2021-11-20T05:36:44.609 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.612+0000 7f1fa96ac700 1 -- 172.21.15.58:0/2507608225 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fa410f450 con 0x7f1fa40f5580 2021-11-20T05:36:44.610 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.612+0000 7f1f9b7fe700 1 -- 172.21.15.58:0/2507608225 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2944 (secure 0 0 0) 0x7f1f94017090 con 0x7f1fa40f5580 2021-11-20T05:36:44.610 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:44.611 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":45},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:35.335786+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14572,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14572":{"gid":14572,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":53,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/444955435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":444955435},{"type":"v1","addr":"172.21.15.58:6827","nonce":444955435}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:44.612 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.614+0000 7f1fa96ac700 1 -- 172.21.15.58:0/2507608225 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1f9005c2a0 msgr2=0x7f1f9005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:44.613 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.614+0000 7f1fa96ac700 1 --2- 172.21.15.58:0/2507608225 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1f9005c2a0 0x7f1f9005e760 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f1f8c004630 tx=0x7f1f8c009b30).stop 2021-11-20T05:36:44.613 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.614+0000 7f1fa96ac700 1 -- 172.21.15.58:0/2507608225 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1fa40f5580 msgr2=0x7f1fa40f9500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:44.613 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.614+0000 7f1fa96ac700 1 --2- 172.21.15.58:0/2507608225 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1fa40f5580 0x7f1fa40f9500 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f1f9400c530 tx=0x7f1f94007e10).stop 2021-11-20T05:36:44.613 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.614+0000 7f1fa96ac700 1 -- 172.21.15.58:0/2507608225 shutdown_connections 2021-11-20T05:36:44.613 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.614+0000 7f1fa96ac700 1 --2- 172.21.15.58:0/2507608225 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1f9005c2a0 0x7f1f9005e760 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:44.614 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.614+0000 7f1fa96ac700 1 --2- 172.21.15.58:0/2507608225 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1fa40f5580 0x7f1fa40f9500 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:44.614 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.614+0000 7f1fa96ac700 1 --2- 172.21.15.58:0/2507608225 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1fa40f6820 0x7f1fa40f7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:44.614 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.614+0000 7f1fa96ac700 1 -- 172.21.15.58:0/2507608225 >> 172.21.15.58:0/2507608225 conn(0x7f1fa40f0ce0 msgr2=0x7f1fa4060240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:44.614 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.614+0000 7f1fa96ac700 1 -- 172.21.15.58:0/2507608225 shutdown_connections 2021-11-20T05:36:44.615 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:44.614+0000 7f1fa96ac700 1 -- 172.21.15.58:0/2507608225 wait complete. 2021-11-20T05:36:44.615 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 57 2021-11-20T05:36:44.624 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:35.335786+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14572, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14572': {'gid': 14572, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 53, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/444955435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 444955435}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 444955435}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:44.624 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14572:up:rejoin 2021-11-20T05:36:45.625 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:45.650 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:45 smithi058 conmon[33537]: cluster 2021-11-20T05:36:44.268578+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:36:45.651 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:45 smithi058 conmon[33537]: 271 : cluster [DBG] pgmap v241: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:36:45.651 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:45 smithi058 conmon[33537]: audit 2021-11-20T05:36:44.613401+0000 mon.smithi058 (mon.0) 676 : audit [DBG] from='client.? 172.21.15.58:0/2507608225' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:45.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:45 smithi187 conmon[36358]: cluster 2021-11-20T05:36:44.268578+0000 mgr.smithi058.zagdlk (mgr.14164) 271 : cluster [DBG] pgmap v241: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:36:45.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:45 smithi187 conmon[36358]: audit 2021-11-20T05:36:44.613401+0000 mon.smithi058 (mon.0) 676 : audit [DBG] from='client.? 172.21.15.58:0/2507608225' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:45.777 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.776+0000 7fbd741e2700 1 -- 172.21.15.58:0/3072857366 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd6c0f6840 msgr2=0x7fbd6c0f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:45.778 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.776+0000 7fbd741e2700 1 --2- 172.21.15.58:0/3072857366 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd6c0f6840 0x7fbd6c0f6ca0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fbd5c004660 tx=0x7fbd5c009b30).stop 2021-11-20T05:36:45.778 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.776+0000 7fbd741e2700 1 -- 172.21.15.58:0/3072857366 shutdown_connections 2021-11-20T05:36:45.778 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.776+0000 7fbd741e2700 1 --2- 172.21.15.58:0/3072857366 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd6c0f6840 0x7fbd6c0f6ca0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:45.778 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.776+0000 7fbd741e2700 1 --2- 172.21.15.58:0/3072857366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd6c0f55a0 0x7fbd6c0f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:45.779 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.776+0000 7fbd741e2700 1 -- 172.21.15.58:0/3072857366 >> 172.21.15.58:0/3072857366 conn(0x7fbd6c0f0ce0 msgr2=0x7fbd6c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:45.779 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.777+0000 7fbd741e2700 1 -- 172.21.15.58:0/3072857366 shutdown_connections 2021-11-20T05:36:45.779 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.777+0000 7fbd741e2700 1 -- 172.21.15.58:0/3072857366 wait complete. 2021-11-20T05:36:45.780 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.777+0000 7fbd741e2700 1 Processor -- start 2021-11-20T05:36:45.780 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.777+0000 7fbd741e2700 1 -- start start 2021-11-20T05:36:45.780 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.777+0000 7fbd741e2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd6c0f55a0 0x7fbd6c05ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:45.780 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.777+0000 7fbd741e2700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd6c0f6840 0x7fbd6c10bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:45.781 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.777+0000 7fbd741e2700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c10c180 con 0x7fbd6c0f6840 2021-11-20T05:36:45.781 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.777+0000 7fbd741e2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c10c2c0 con 0x7fbd6c0f55a0 2021-11-20T05:36:45.781 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.777+0000 7fbd7177d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd6c0f6840 0x7fbd6c10bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:45.781 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.777+0000 7fbd7177d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd6c0f6840 0x7fbd6c10bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41270/0 (socket says 172.21.15.58:41270) 2021-11-20T05:36:45.781 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.777+0000 7fbd7177d700 1 -- 172.21.15.58:0/2160639111 learned_addr learned my addr 172.21.15.58:0/2160639111 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:45.782 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.777+0000 7fbd71f7e700 1 --2- 172.21.15.58:0/2160639111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd6c0f55a0 0x7fbd6c05ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:45.782 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.778+0000 7fbd7177d700 1 -- 172.21.15.58:0/2160639111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd6c0f55a0 msgr2=0x7fbd6c05ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:45.782 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.778+0000 7fbd7177d700 1 --2- 172.21.15.58:0/2160639111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd6c0f55a0 0x7fbd6c05ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:45.782 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.778+0000 7fbd7177d700 1 -- 172.21.15.58:0/2160639111 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c005040 con 0x7fbd6c0f6840 2021-11-20T05:36:45.783 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.778+0000 7fbd7177d700 1 --2- 172.21.15.58:0/2160639111 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd6c0f6840 0x7fbd6c10bc90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fbd5c009830 tx=0x7fbd5c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:45.783 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.778+0000 7fbd627fc700 1 -- 172.21.15.58:0/2160639111 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd5c018070 con 0x7fbd6c0f6840 2021-11-20T05:36:45.783 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.778+0000 7fbd627fc700 1 -- 172.21.15.58:0/2160639111 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd5c013410 con 0x7fbd6c0f6840 2021-11-20T05:36:45.783 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.778+0000 7fbd741e2700 1 -- 172.21.15.58:0/2160639111 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6c10c4c0 con 0x7fbd6c0f6840 2021-11-20T05:36:45.783 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.778+0000 7fbd627fc700 1 -- 172.21.15.58:0/2160639111 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd5c01c3c0 con 0x7fbd6c0f6840 2021-11-20T05:36:45.784 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.778+0000 7fbd741e2700 1 -- 172.21.15.58:0/2160639111 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6c10ca20 con 0x7fbd6c0f6840 2021-11-20T05:36:45.784 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.779+0000 7fbd627fc700 1 -- 172.21.15.58:0/2160639111 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbd5c01d420 con 0x7fbd6c0f6840 2021-11-20T05:36:45.784 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.779+0000 7fbd741e2700 1 -- 172.21.15.58:0/2160639111 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd6c027a00 con 0x7fbd6c0f6840 2021-11-20T05:36:45.784 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.780+0000 7fbd627fc700 1 --2- 172.21.15.58:0/2160639111 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbd5805c2a0 0x7fbd5805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:45.785 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.780+0000 7fbd627fc700 1 -- 172.21.15.58:0/2160639111 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fbd5c077df0 con 0x7fbd6c0f6840 2021-11-20T05:36:45.785 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.780+0000 7fbd71f7e700 1 --2- 172.21.15.58:0/2160639111 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbd5805c2a0 0x7fbd5805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:45.785 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.780+0000 7fbd71f7e700 1 --2- 172.21.15.58:0/2160639111 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbd5805c2a0 0x7fbd5805e760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fbd6c0f66a0 tx=0x7fbd6800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:45.786 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.784+0000 7fbd627fc700 1 -- 172.21.15.58:0/2160639111 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd5c04a800 con 0x7fbd6c0f6840 2021-11-20T05:36:45.983 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.986+0000 7fbd741e2700 1 -- 172.21.15.58:0/2160639111 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd6c0fdd30 con 0x7fbd6c0f6840 2021-11-20T05:36:45.984 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.987+0000 7fbd627fc700 1 -- 172.21.15.58:0/2160639111 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2944 (secure 0 0 0) 0x7fbd5c04de20 con 0x7fbd6c0f6840 2021-11-20T05:36:45.985 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:45.985 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":45},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:35.335786+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14572,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14572":{"gid":14572,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":53,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/444955435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":444955435},{"type":"v1","addr":"172.21.15.58:6827","nonce":444955435}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:45.986 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.988+0000 7fbd741e2700 1 -- 172.21.15.58:0/2160639111 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbd5805c2a0 msgr2=0x7fbd5805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:45.986 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.988+0000 7fbd741e2700 1 --2- 172.21.15.58:0/2160639111 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbd5805c2a0 0x7fbd5805e760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fbd6c0f66a0 tx=0x7fbd6800b040).stop 2021-11-20T05:36:45.987 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.988+0000 7fbd741e2700 1 -- 172.21.15.58:0/2160639111 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd6c0f6840 msgr2=0x7fbd6c10bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:45.987 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.988+0000 7fbd741e2700 1 --2- 172.21.15.58:0/2160639111 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd6c0f6840 0x7fbd6c10bc90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fbd5c009830 tx=0x7fbd5c009b30).stop 2021-11-20T05:36:45.987 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.988+0000 7fbd741e2700 1 -- 172.21.15.58:0/2160639111 shutdown_connections 2021-11-20T05:36:45.987 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.988+0000 7fbd741e2700 1 --2- 172.21.15.58:0/2160639111 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbd5805c2a0 0x7fbd5805e760 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:45.988 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.988+0000 7fbd741e2700 1 --2- 172.21.15.58:0/2160639111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd6c0f55a0 0x7fbd6c05ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:45.988 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.988+0000 7fbd741e2700 1 --2- 172.21.15.58:0/2160639111 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbd6c0f6840 0x7fbd6c10bc90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:45.988 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.989+0000 7fbd741e2700 1 -- 172.21.15.58:0/2160639111 >> 172.21.15.58:0/2160639111 conn(0x7fbd6c0f0ce0 msgr2=0x7fbd6c0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:45.988 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.989+0000 7fbd741e2700 1 -- 172.21.15.58:0/2160639111 shutdown_connections 2021-11-20T05:36:45.988 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:45.989+0000 7fbd741e2700 1 -- 172.21.15.58:0/2160639111 wait complete. 2021-11-20T05:36:45.989 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 57 2021-11-20T05:36:45.998 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:35.335786+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14572, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14572': {'gid': 14572, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 53, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/444955435', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 444955435}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 444955435}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:45.998 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14572:up:rejoin 2021-11-20T05:36:46.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:46 smithi187 conmon[36358]: audit 2021-11-20T05:36:45.987679+0000 mon.smithi058 (mon.0) 677 : audit [DBG] 2021-11-20T05:36:46.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:46 smithi187 conmon[36358]: from='client.? 172.21.15.58:0/2160639111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:46.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:46 smithi187 conmon[36358]: cluster 2021-11-20T05:36:46.250240+0000 mon.smithi058 (mon.0) 678 : cluster [INF] daemon mds.cephfs.smithi058.rxapzd restarted 2021-11-20T05:36:46.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:46 smithi187 conmon[36358]: cluster 2021-11-20T05:36:46.253938+0000 mon.smithi058 (mon.0) 679 : cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2021-11-20T05:36:46.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:46 smithi058 conmon[33537]: audit 2021-11-20T05:36:45.987679+0000 mon.smithi058 (mon.0) 677 : audit [DBG] from='client.? 172.21.15.58:0/2160639111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:46.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:46 smithi058 conmon[33537]: cluster 2021-11-20T05:36:46.250240+0000 mon.smithi058 (mon.0) 678 : cluster [INF] daemon mds.cephfs.smithi058.rxapzd restarted 2021-11-20T05:36:46.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:46 smithi058 conmon[33537]: cluster 2021-11-20T05:36:46.253938+0000 mon.smithi058 (mon.0) 679 : cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2021-11-20T05:36:46.999 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:47.135 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.136+0000 7f0bf502d700 1 -- 172.21.15.58:0/1096151905 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0bf00f5580 msgr2=0x7f0bf00f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:47.136 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.136+0000 7f0bf502d700 1 --2- 172.21.15.58:0/1096151905 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0bf00f5580 0x7f0bf00f59a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f0be0004660 tx=0x7f0be0009b30).stop 2021-11-20T05:36:47.136 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.136+0000 7f0bf502d700 1 -- 172.21.15.58:0/1096151905 shutdown_connections 2021-11-20T05:36:47.136 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.136+0000 7f0bf502d700 1 --2- 172.21.15.58:0/1096151905 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0bf00f6820 0x7f0bf00f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:47.136 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.136+0000 7f0bf502d700 1 --2- 172.21.15.58:0/1096151905 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0bf00f5580 0x7f0bf00f59a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:47.137 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.136+0000 7f0bf502d700 1 -- 172.21.15.58:0/1096151905 >> 172.21.15.58:0/1096151905 conn(0x7f0bf00f0ce0 msgr2=0x7f0bf00f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:47.137 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.136+0000 7f0bf502d700 1 -- 172.21.15.58:0/1096151905 shutdown_connections 2021-11-20T05:36:47.137 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.136+0000 7f0bf502d700 1 -- 172.21.15.58:0/1096151905 wait complete. 2021-11-20T05:36:47.138 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.137+0000 7f0bf502d700 1 Processor -- start 2021-11-20T05:36:47.138 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.137+0000 7f0bf502d700 1 -- start start 2021-11-20T05:36:47.138 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.137+0000 7f0bf502d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0bf00f5580 0x7f0bf0106e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:47.138 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.137+0000 7f0bf502d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0bf00f6820 0x7f0bf0107310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:47.139 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.137+0000 7f0bf502d700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf01078c0 con 0x7f0bf00f5580 2021-11-20T05:36:47.139 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.137+0000 7f0bf502d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf0107a00 con 0x7f0bf00f6820 2021-11-20T05:36:47.139 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.137+0000 7f0beed9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0bf00f5580 0x7f0bf0106e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:47.139 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.137+0000 7f0beed9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0bf00f5580 0x7f0bf0106e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41288/0 (socket says 172.21.15.58:41288) 2021-11-20T05:36:47.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.137+0000 7f0beed9d700 1 -- 172.21.15.58:0/1757914054 learned_addr learned my addr 172.21.15.58:0/1757914054 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:47.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.137+0000 7f0bee59c700 1 --2- 172.21.15.58:0/1757914054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0bf00f6820 0x7f0bf0107310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:47.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.137+0000 7f0beed9d700 1 -- 172.21.15.58:0/1757914054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0bf00f6820 msgr2=0x7f0bf0107310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:47.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.137+0000 7f0beed9d700 1 --2- 172.21.15.58:0/1757914054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0bf00f6820 0x7f0bf0107310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:47.141 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.138+0000 7f0beed9d700 1 -- 172.21.15.58:0/1757914054 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be0005040 con 0x7f0bf00f5580 2021-11-20T05:36:47.141 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.138+0000 7f0beed9d700 1 --2- 172.21.15.58:0/1757914054 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0bf00f5580 0x7f0bf0106e20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f0be0004e30 tx=0x7f0be0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:47.141 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.138+0000 7f0be77fe700 1 -- 172.21.15.58:0/1757914054 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0be0018070 con 0x7f0bf00f5580 2021-11-20T05:36:47.142 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.138+0000 7f0bf502d700 1 -- 172.21.15.58:0/1757914054 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf010c440 con 0x7f0bf00f5580 2021-11-20T05:36:47.142 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.138+0000 7f0be77fe700 1 -- 172.21.15.58:0/1757914054 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0be0013410 con 0x7f0bf00f5580 2021-11-20T05:36:47.142 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.138+0000 7f0be77fe700 1 -- 172.21.15.58:0/1757914054 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0be001c3c0 con 0x7f0bf00f5580 2021-11-20T05:36:47.142 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.138+0000 7f0bf502d700 1 -- 172.21.15.58:0/1757914054 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf010ca00 con 0x7f0bf00f5580 2021-11-20T05:36:47.144 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.139+0000 7f0bf502d700 1 -- 172.21.15.58:0/1757914054 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf00fdd10 con 0x7f0bf00f5580 2021-11-20T05:36:47.144 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.139+0000 7f0be77fe700 1 -- 172.21.15.58:0/1757914054 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0be001d420 con 0x7f0bf00f5580 2021-11-20T05:36:47.144 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.140+0000 7f0be77fe700 1 --2- 172.21.15.58:0/1757914054 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0bdc05c2a0 0x7f0bdc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:47.145 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.140+0000 7f0be77fe700 1 -- 172.21.15.58:0/1757914054 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f0be0077dc0 con 0x7f0bf00f5580 2021-11-20T05:36:47.145 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.140+0000 7f0bee59c700 1 --2- 172.21.15.58:0/1757914054 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0bdc05c2a0 0x7f0bdc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:47.145 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.140+0000 7f0bee59c700 1 --2- 172.21.15.58:0/1757914054 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0bdc05c2a0 0x7f0bdc05e760 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f0bd8009fd0 tx=0x7f0bd800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:47.145 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.144+0000 7f0be77fe700 1 -- 172.21.15.58:0/1757914054 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0be004a780 con 0x7f0bf00f5580 2021-11-20T05:36:47.341 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.344+0000 7f0bf502d700 1 -- 172.21.15.58:0/1757914054 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bf010d040 con 0x7f0bf00f5580 2021-11-20T05:36:47.342 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.345+0000 7f0be77fe700 1 -- 172.21.15.58:0/1757914054 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+2950 (secure 0 0 0) 0x7f0be004dda0 con 0x7f0bf00f5580 2021-11-20T05:36:47.342 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:47.343 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:47.264321+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24367,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24367":{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":59,"state":"up:resolve","state_seq":13,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:47.344 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.346+0000 7f0bf502d700 1 -- 172.21.15.58:0/1757914054 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0bdc05c2a0 msgr2=0x7f0bdc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:47.344 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.346+0000 7f0bf502d700 1 --2- 172.21.15.58:0/1757914054 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0bdc05c2a0 0x7f0bdc05e760 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f0bd8009fd0 tx=0x7f0bd800b040).stop 2021-11-20T05:36:47.344 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.346+0000 7f0bf502d700 1 -- 172.21.15.58:0/1757914054 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0bf00f5580 msgr2=0x7f0bf0106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:47.345 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.346+0000 7f0bf502d700 1 --2- 172.21.15.58:0/1757914054 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0bf00f5580 0x7f0bf0106e20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f0be0004e30 tx=0x7f0be0009b30).stop 2021-11-20T05:36:47.345 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.346+0000 7f0bf502d700 1 -- 172.21.15.58:0/1757914054 shutdown_connections 2021-11-20T05:36:47.345 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.346+0000 7f0bf502d700 1 --2- 172.21.15.58:0/1757914054 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0bdc05c2a0 0x7f0bdc05e760 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:47.345 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.346+0000 7f0bf502d700 1 --2- 172.21.15.58:0/1757914054 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0bf00f5580 0x7f0bf0106e20 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:47.345 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.346+0000 7f0bf502d700 1 --2- 172.21.15.58:0/1757914054 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0bf00f6820 0x7f0bf0107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:47.346 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.346+0000 7f0bf502d700 1 -- 172.21.15.58:0/1757914054 >> 172.21.15.58:0/1757914054 conn(0x7f0bf00f0ce0 msgr2=0x7f0bf0060240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:47.346 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.346+0000 7f0bf502d700 1 -- 172.21.15.58:0/1757914054 shutdown_connections 2021-11-20T05:36:47.346 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:47.346+0000 7f0bf502d700 1 -- 172.21.15.58:0/1757914054 wait complete. 2021-11-20T05:36:47.346 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 60 2021-11-20T05:36:47.354 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:47 smithi058 conmon[33537]: cluster 2021-11-20T05:36:46.261808+0000 2021-11-20T05:36:47.355 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:47 smithi058 conmon[33537]: mon.smithi058 (mon.0) 680 : cluster [DBG] mds.? [v2:172.21.15.58:6826/3191333321,v1:172.21.15.58:6827/3191333321] up:boot 2021-11-20T05:36:47.355 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:47 smithi058 conmon[33537]: cluster 2021-11-20T05:36:46.261888+0000 mon.smithi058 (mon.0) 681 : cluster [INF] Standby daemon mds.cephfs.smithi187.wxduhu assigned to filesystem cephfs as rank 0 2021-11-20T05:36:47.355 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:47 smithi058 conmon[33537]: cluster 2021-11-20T05:36:46.262142+0000 mon.smithi058 (mon.0) 682 : 2021-11-20T05:36:47.355 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:47 smithi058 conmon[33537]: cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 3 up:standby, 1 failed 2021-11-20T05:36:47.356 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:47 smithi058 conmon[33537]: audit 2021-11-20 2021-11-20T05:36:47.356 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:47 smithi058 conmon[33537]: T05:36:46.262338+0000 mon.smithi058 (mon. 2021-11-20T05:36:47.356 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:47 smithi058 conmon[33537]: 0) 683 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi058.rxapzd"}]: dispatch 2021-11-20T05:36:47.356 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:47 smithi058 conmon[33537]: cluster 2021-11-20T05:36:46.264312+0000 2021-11-20T05:36:47.356 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:47 smithi058 conmon[33537]: mon.smithi058 (mon.0) 684 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:47.357 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:47 smithi058 conmon[33537]: cluster 2021-11-20T05:36:46. 2021-11-20T05:36:47.357 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:47 smithi058 conmon[33537]: 268941+0000 mgr.smithi058.zagdlk (mgr.14164) 272 : cluster [DBG] pgmap v243: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T05:36:47.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:47.264321+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24367, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24367': {'gid': 24367, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 59, 'state': 'up:resolve', 'state_seq': 13, 'addr': '172.21.15.187:6825/126010257', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 126010257}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 126010257}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:47.358 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24367:up:resolve 2021-11-20T05:36:47.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:47 smithi187 conmon[36358]: cluster 2021-11-20T05:36:46.261808+0000 mon.smithi058 (mon.0) 680 : cluster [DBG] mds.? [v2:172.21.15.58:6826/3191333321,v1:172.21.15.58:6827/3191333321] up:boot 2021-11-20T05:36:47.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:47 smithi187 conmon[36358]: cluster 2021-11-20T 2021-11-20T05:36:47.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:47 smithi187 conmon[36358]: 05:36:46.261888+0000 mon.smithi058 (mon.0) 681 : cluster [INF] Standby daemon mds.cephfs.smithi187.wxduhu assigned to filesystem cephfs as rank 0 2021-11-20T05:36:47.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:47 smithi187 conmon[36358]: cluster 2021-11-20T05:36:46.262142+0000 mon.smithi058 (mon.0) 682 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 3 up:standby, 1 failed 2021-11-20T05:36:47.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:47 smithi187 conmon[36358]: audit 2021 2021-11-20T05:36:47.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:47 smithi187 conmon[36358]: -11-20T05:36:46.262338+0000 mon.smithi058 (mon.0) 683 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi058.rxapzd"}]: dispatch 2021-11-20T05:36:47.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:47 smithi187 conmon[36358]: cluster 2021- 2021-11-20T05:36:47.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:47 smithi187 conmon[36358]: 11-20T05:36:46.264312+0000 mon.smithi058 ( 2021-11-20T05:36:47.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:47 smithi187 conmon[36358]: mon.0) 684 : cluster [DBG] 2021-11-20T05:36:47.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:47 smithi187 conmon[36358]: fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:47.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:47 smithi187 conmon[36358]: cluster 2021- 2021-11-20T05:36:47.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:47 smithi187 conmon[36358]: 11-20T05:36:46.268941+0000 2021-11-20T05:36:47.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:47 smithi187 conmon[36358]: mgr.smithi058.zagdlk (mgr.14164) 272 : cluster 2021-11-20T05:36:47.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:47 smithi187 conmon[36358]: [DBG] pgmap v243: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T05:36:48.360 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:48.491 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.491+0000 7f4c64379700 1 -- 172.21.15.58:0/1152700685 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4c5c0f5300 msgr2=0x7f4c5c0f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:48.491 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.491+0000 7f4c64379700 1 --2- 172.21.15.58:0/1152700685 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4c5c0f5300 0x7f4c5c0f7a80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f4c58004660 tx=0x7f4c58009b30).stop 2021-11-20T05:36:48.492 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.491+0000 7f4c64379700 1 -- 172.21.15.58:0/1152700685 shutdown_connections 2021-11-20T05:36:48.492 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.491+0000 7f4c64379700 1 --2- 172.21.15.58:0/1152700685 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4c5c0f5300 0x7f4c5c0f7a80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:48.492 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.491+0000 7f4c64379700 1 --2- 172.21.15.58:0/1152700685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c5c0f4960 0x7f4c5c0f4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:48.492 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.491+0000 7f4c64379700 1 -- 172.21.15.58:0/1152700685 >> 172.21.15.58:0/1152700685 conn(0x7f4c5c0f0440 msgr2=0x7f4c5c0f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:48.493 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.491+0000 7f4c64379700 1 -- 172.21.15.58:0/1152700685 shutdown_connections 2021-11-20T05:36:48.493 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.491+0000 7f4c64379700 1 -- 172.21.15.58:0/1152700685 wait complete. 2021-11-20T05:36:48.493 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.492+0000 7f4c64379700 1 Processor -- start 2021-11-20T05:36:48.493 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.492+0000 7f4c64379700 1 -- start start 2021-11-20T05:36:48.494 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.492+0000 7f4c64379700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4c5c0f4960 0x7f4c5c106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:48.494 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.492+0000 7f4c64379700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c5c0f5300 0x7f4c5c1073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:48.494 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.492+0000 7f4c64379700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c107970 con 0x7f4c5c0f4960 2021-11-20T05:36:48.494 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.492+0000 7f4c64379700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c107ab0 con 0x7f4c5c0f5300 2021-11-20T05:36:48.495 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.492+0000 7f4c62115700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4c5c0f4960 0x7f4c5c106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:48.495 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.492+0000 7f4c62115700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4c5c0f4960 0x7f4c5c106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41300/0 (socket says 172.21.15.58:41300) 2021-11-20T05:36:48.495 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.492+0000 7f4c62115700 1 -- 172.21.15.58:0/2821711200 learned_addr learned my addr 172.21.15.58:0/2821711200 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:48.495 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.492+0000 7f4c61914700 1 --2- 172.21.15.58:0/2821711200 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c5c0f5300 0x7f4c5c1073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:48.495 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.492+0000 7f4c62115700 1 -- 172.21.15.58:0/2821711200 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c5c0f5300 msgr2=0x7f4c5c1073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:48.496 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.492+0000 7f4c62115700 1 --2- 172.21.15.58:0/2821711200 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c5c0f5300 0x7f4c5c1073c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:48.496 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.492+0000 7f4c62115700 1 -- 172.21.15.58:0/2821711200 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c58005040 con 0x7f4c5c0f4960 2021-11-20T05:36:48.496 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.492+0000 7f4c61914700 1 --2- 172.21.15.58:0/2821711200 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c5c0f5300 0x7f4c5c1073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:36:48.496 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.493+0000 7f4c62115700 1 --2- 172.21.15.58:0/2821711200 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4c5c0f4960 0x7f4c5c106ed0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f4c4c00c530 tx=0x7f4c4c007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:48.497 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.493+0000 7f4c52ffd700 1 -- 172.21.15.58:0/2821711200 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c4c00f650 con 0x7f4c5c0f4960 2021-11-20T05:36:48.497 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.493+0000 7f4c52ffd700 1 -- 172.21.15.58:0/2821711200 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c4c00fc60 con 0x7f4c5c0f4960 2021-11-20T05:36:48.497 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.493+0000 7f4c64379700 1 -- 172.21.15.58:0/2821711200 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5c10c550 con 0x7f4c5c0f4960 2021-11-20T05:36:48.497 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.493+0000 7f4c52ffd700 1 -- 172.21.15.58:0/2821711200 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c4c012d20 con 0x7f4c5c0f4960 2021-11-20T05:36:48.497 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.493+0000 7f4c64379700 1 -- 172.21.15.58:0/2821711200 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c5c10cb10 con 0x7f4c5c0f4960 2021-11-20T05:36:48.498 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.494+0000 7f4c64379700 1 -- 172.21.15.58:0/2821711200 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c5c027a00 con 0x7f4c5c0f4960 2021-11-20T05:36:48.498 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.494+0000 7f4c52ffd700 1 -- 172.21.15.58:0/2821711200 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4c4c019030 con 0x7f4c5c0f4960 2021-11-20T05:36:48.499 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.494+0000 7f4c52ffd700 1 --2- 172.21.15.58:0/2821711200 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4c4805c2a0 0x7f4c4805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:48.499 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.495+0000 7f4c52ffd700 1 -- 172.21.15.58:0/2821711200 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f4c4c075de0 con 0x7f4c5c0f4960 2021-11-20T05:36:48.499 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.495+0000 7f4c61914700 1 --2- 172.21.15.58:0/2821711200 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4c4805c2a0 0x7f4c4805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:48.500 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.495+0000 7f4c61914700 1 --2- 172.21.15.58:0/2821711200 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4c4805c2a0 0x7f4c4805e760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f4c58009d60 tx=0x7f4c58009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:48.500 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.498+0000 7f4c52ffd700 1 -- 172.21.15.58:0/2821711200 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c4c0462b0 con 0x7f4c5c0f4960 2021-11-20T05:36:48.690 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.693+0000 7f4c64379700 1 -- 172.21.15.58:0/2821711200 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c5c0fee50 con 0x7f4c5c0f4960 2021-11-20T05:36:48.691 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.693+0000 7f4c52ffd700 1 -- 172.21.15.58:0/2821711200 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2952 (secure 0 0 0) 0x7f4c4c017020 con 0x7f4c5c0f4960 2021-11-20T05:36:48.691 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:48.692 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:48.268827+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24367,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24367":{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":59,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:48.693 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.695+0000 7f4c64379700 1 -- 172.21.15.58:0/2821711200 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4c4805c2a0 msgr2=0x7f4c4805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:48.693 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.695+0000 7f4c64379700 1 --2- 172.21.15.58:0/2821711200 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4c4805c2a0 0x7f4c4805e760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f4c58009d60 tx=0x7f4c58009b30).stop 2021-11-20T05:36:48.693 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.695+0000 7f4c64379700 1 -- 172.21.15.58:0/2821711200 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4c5c0f4960 msgr2=0x7f4c5c106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:48.694 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.695+0000 7f4c64379700 1 --2- 172.21.15.58:0/2821711200 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4c5c0f4960 0x7f4c5c106ed0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f4c4c00c530 tx=0x7f4c4c007e10).stop 2021-11-20T05:36:48.694 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.695+0000 7f4c64379700 1 -- 172.21.15.58:0/2821711200 shutdown_connections 2021-11-20T05:36:48.694 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.695+0000 7f4c64379700 1 --2- 172.21.15.58:0/2821711200 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4c4805c2a0 0x7f4c4805e760 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:48.694 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.695+0000 7f4c64379700 1 --2- 172.21.15.58:0/2821711200 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4c5c0f4960 0x7f4c5c106ed0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:48.695 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.695+0000 7f4c64379700 1 --2- 172.21.15.58:0/2821711200 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4c5c0f5300 0x7f4c5c1073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:48.695 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.695+0000 7f4c64379700 1 -- 172.21.15.58:0/2821711200 >> 172.21.15.58:0/2821711200 conn(0x7f4c5c0f0440 msgr2=0x7f4c5c0fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:48.695 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.695+0000 7f4c64379700 1 -- 172.21.15.58:0/2821711200 shutdown_connections 2021-11-20T05:36:48.695 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:48.695+0000 7f4c64379700 1 -- 172.21.15.58:0/2821711200 wait complete. 2021-11-20T05:36:48.696 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 61 2021-11-20T05:36:48.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:48 smithi187 conmon[36358]: cluster 2021-11-20T05:36:47.269688+0000 mon.smithi058 (mon.0) 685 : 2021-11-20T05:36:48.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:48 smithi187 conmon[36358]: cluster [DBG] mds.? [v2:172.21.15.187:6824/126010257,v1:172.21.15.187:6825/126010257] up:resolve 2021-11-20T05:36:48.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:48 smithi187 conmon[36358]: cluster 2021-11-20T05:36:47.269765+0000 mon.smithi058 (mon.0) 686 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:48.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:48 smithi187 conmon[36358]: audit 2021-11-20T05:36:47.345587+0000 2021-11-20T05:36:48.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:48 smithi187 conmon[36358]: mon.smithi058 (mon.0) 687 : audit [DBG] from='client.? 172.21.15.58:0/1757914054' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:48.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:48 smithi187 conmon[36358]: 2021-11-20T05:36:48.703 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:48 smithi058 conmon[33537]: cluster 2021-11-20T05:36:47.269688+0000 mon.smithi058 (mon.0 2021-11-20T05:36:48.703 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:48 smithi058 conmon[33537]: ) 685 : cluster [DBG] mds.? [v2:172.21.15.187:6824/126010257,v1:172.21.15.187:6825/126010257] up:resolve 2021-11-20T05:36:48.704 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:48 smithi058 conmon[33537]: cluster 2021-11-20T05:36:47.269765+0000 mon.smithi058 (mon.0) 686 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:48.704 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:48 smithi058 conmon[33537]: audit 2021-11-20T05:36:47.345587+0000 mon.smithi058 2021-11-20T05:36:48.704 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:48 smithi058 conmon[33537]: (mon.0) 687 : audit [DBG] from='client.? 172.21.15.58:0/1757914054' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:48.706 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:48.268827+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24367, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24367': {'gid': 24367, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 59, 'state': 'up:reconnect', 'state_seq': 14, 'addr': '172.21.15.187:6825/126010257', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 126010257}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 126010257}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:48.706 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24367:up:reconnect 2021-11-20T05:36:49.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:49 smithi058 conmon[33537]: cluster 2021-11-20T05:36:48.269386+0000 mgr.smithi058.zagdlk (mgr 2021-11-20T05:36:49.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:49 smithi058 conmon[33537]: .14164) 273 : cluster [DBG] pgmap v244: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 306 B/s rd, 0 op/s 2021-11-20T05:36:49.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:49 smithi058 conmon[33537]: cluster 2021-11-20T05:36:48.275787+0000 mon.smithi058 (mon.0) 688 : cluster [DBG] mds.? [v2:172.21.15.187:6824/126010257,v1:172.21.15.187:6825/126010257] up:reconnect 2021-11-20T05:36:49.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:49 smithi058 conmon[33537]: cluster 2021-11-20T05:36:48.275910+0000 mon.smithi058 (mon.0) 689 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:49.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:49 smithi058 conmon[33537]: audit 2021-11-20T05:36:48.694511+0000 mon.smithi058 (mon.0) 690 : audit [DBG] from='client.? 172.21.15.58:0/2821711200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:49.654 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:49 smithi187 conmon[36358]: cluster 2021-11-20T05:36:48.269386+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:36:49.654 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:49 smithi187 conmon[36358]: 273 : cluster [DBG] pgmap v244: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 306 B/s rd, 0 op/s 2021-11-20T05:36:49.655 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:49 smithi187 conmon[36358]: cluster 2021-11-20T05:36:48.275787+0000 mon.smithi058 (mon.0) 688 : cluster [DBG] mds.? [v2:172.21.15.187:6824/126010257,v1:172.21.15.187:6825/126010257] up:reconnect 2021-11-20T05:36:49.655 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:49 smithi187 conmon[36358]: cluster 2021-11-20T05:36:48.275910+0000 mon.smithi058 (mon.0) 689 : 2021-11-20T05:36:49.655 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:49 smithi187 conmon[36358]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:49.655 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:49 smithi187 conmon[36358]: audit 2021-11-20T05:36 2021-11-20T05:36:49.656 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:49 smithi187 conmon[36358]: :48.694511+0000 mon.smithi058 (mon.0) 690 2021-11-20T05:36:49.656 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:49 smithi187 conmon[36358]: : audit [DBG] from='client.? 172.21.15.58:0/2821711200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:49.707 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:49.843 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.844+0000 7f6d0f7d0700 1 -- 172.21.15.58:0/2840488586 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6d080f6820 msgr2=0x7f6d080f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:49.844 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.844+0000 7f6d0f7d0700 1 --2- 172.21.15.58:0/2840488586 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6d080f6820 0x7f6d080f6c80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f6cfc004660 tx=0x7f6cfc009b30).stop 2021-11-20T05:36:49.844 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.844+0000 7f6d0f7d0700 1 -- 172.21.15.58:0/2840488586 shutdown_connections 2021-11-20T05:36:49.844 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.844+0000 7f6d0f7d0700 1 --2- 172.21.15.58:0/2840488586 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6d080f6820 0x7f6d080f6c80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:49.844 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.844+0000 7f6d0f7d0700 1 --2- 172.21.15.58:0/2840488586 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d080f5580 0x7f6d080f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:49.845 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.844+0000 7f6d0f7d0700 1 -- 172.21.15.58:0/2840488586 >> 172.21.15.58:0/2840488586 conn(0x7f6d080f0ce0 msgr2=0x7f6d080f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:49.845 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.844+0000 7f6d0f7d0700 1 -- 172.21.15.58:0/2840488586 shutdown_connections 2021-11-20T05:36:49.845 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.844+0000 7f6d0f7d0700 1 -- 172.21.15.58:0/2840488586 wait complete. 2021-11-20T05:36:49.845 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.844+0000 7f6d0f7d0700 1 Processor -- start 2021-11-20T05:36:49.845 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.845+0000 7f6d0f7d0700 1 -- start start 2021-11-20T05:36:49.846 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.845+0000 7f6d0f7d0700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6d080f5580 0x7f6d080f9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:49.846 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.845+0000 7f6d0f7d0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d080f6820 0x7f6d080f8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:49.846 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.845+0000 7f6d0f7d0700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080fa2d0 con 0x7f6d080f5580 2021-11-20T05:36:49.846 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.845+0000 7f6d0f7d0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080f8870 con 0x7f6d080f6820 2021-11-20T05:36:49.847 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.845+0000 7f6d0d56c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6d080f5580 0x7f6d080f9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:49.847 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.845+0000 7f6d0d56c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6d080f5580 0x7f6d080f9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41314/0 (socket says 172.21.15.58:41314) 2021-11-20T05:36:49.847 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.845+0000 7f6d0d56c700 1 -- 172.21.15.58:0/1136520446 learned_addr learned my addr 172.21.15.58:0/1136520446 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:49.847 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.845+0000 7f6d0cd6b700 1 --2- 172.21.15.58:0/1136520446 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d080f6820 0x7f6d080f8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:49.848 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.845+0000 7f6d0d56c700 1 -- 172.21.15.58:0/1136520446 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d080f6820 msgr2=0x7f6d080f8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:49.848 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.845+0000 7f6d0d56c700 1 --2- 172.21.15.58:0/1136520446 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d080f6820 0x7f6d080f8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:49.848 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.845+0000 7f6d0d56c700 1 -- 172.21.15.58:0/1136520446 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cfc005040 con 0x7f6d080f5580 2021-11-20T05:36:49.848 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.845+0000 7f6d0d56c700 1 --2- 172.21.15.58:0/1136520446 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6d080f5580 0x7f6d080f9d20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f6d04002700 tx=0x7f6d04009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:49.849 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.845+0000 7f6cf9ffb700 1 -- 172.21.15.58:0/1136520446 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d040079c0 con 0x7f6d080f5580 2021-11-20T05:36:49.849 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.845+0000 7f6cf9ffb700 1 -- 172.21.15.58:0/1136520446 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d0400f040 con 0x7f6d080f5580 2021-11-20T05:36:49.849 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.845+0000 7f6d0f7d0700 1 -- 172.21.15.58:0/1136520446 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d080f8af0 con 0x7f6d080f5580 2021-11-20T05:36:49.849 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.845+0000 7f6cf9ffb700 1 -- 172.21.15.58:0/1136520446 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d04014860 con 0x7f6d080f5580 2021-11-20T05:36:49.850 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.846+0000 7f6d0f7d0700 1 -- 172.21.15.58:0/1136520446 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d080f90b0 con 0x7f6d080f5580 2021-11-20T05:36:49.851 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.847+0000 7f6d0f7d0700 1 -- 172.21.15.58:0/1136520446 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d080fdd10 con 0x7f6d080f5580 2021-11-20T05:36:49.851 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.847+0000 7f6cf9ffb700 1 -- 172.21.15.58:0/1136520446 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6d0401b060 con 0x7f6d080f5580 2021-11-20T05:36:49.851 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.847+0000 7f6cf9ffb700 1 --2- 172.21.15.58:0/1136520446 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6cf405c2a0 0x7f6cf405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:49.852 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.847+0000 7f6cf9ffb700 1 -- 172.21.15.58:0/1136520446 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f6d04075fd0 con 0x7f6d080f5580 2021-11-20T05:36:49.852 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.847+0000 7f6d0cd6b700 1 --2- 172.21.15.58:0/1136520446 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6cf405c2a0 0x7f6cf405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:49.852 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.848+0000 7f6d0cd6b700 1 --2- 172.21.15.58:0/1136520446 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6cf405c2a0 0x7f6cf405e760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f6cfc005010 tx=0x7f6cfc009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:49.852 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:49.851+0000 7f6cf9ffb700 1 -- 172.21.15.58:0/1136520446 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d04048a10 con 0x7f6d080f5580 2021-11-20T05:36:50.044 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:50.046+0000 7f6d0f7d0700 1 -- 172.21.15.58:0/1136520446 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d0810f520 con 0x7f6d080f5580 2021-11-20T05:36:50.044 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:50.047+0000 7f6cf9ffb700 1 -- 172.21.15.58:0/1136520446 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2949 (secure 0 0 0) 0x7f6d0404c030 con 0x7f6d080f5580 2021-11-20T05:36:50.045 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:50.045 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:49.273931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24367,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24367":{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":59,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:50.047 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:50.049+0000 7f6d0f7d0700 1 -- 172.21.15.58:0/1136520446 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6cf405c2a0 msgr2=0x7f6cf405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:50.048 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:50.049+0000 7f6d0f7d0700 1 --2- 172.21.15.58:0/1136520446 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6cf405c2a0 0x7f6cf405e760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f6cfc005010 tx=0x7f6cfc009b30).stop 2021-11-20T05:36:50.048 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:50.049+0000 7f6d0f7d0700 1 -- 172.21.15.58:0/1136520446 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6d080f5580 msgr2=0x7f6d080f9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:50.049 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:50.049+0000 7f6d0f7d0700 1 --2- 172.21.15.58:0/1136520446 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6d080f5580 0x7f6d080f9d20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f6d04002700 tx=0x7f6d04009d80).stop 2021-11-20T05:36:50.049 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:50.049+0000 7f6d0f7d0700 1 -- 172.21.15.58:0/1136520446 shutdown_connections 2021-11-20T05:36:50.049 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:50.049+0000 7f6d0f7d0700 1 --2- 172.21.15.58:0/1136520446 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6cf405c2a0 0x7f6cf405e760 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:50.049 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:50.049+0000 7f6d0f7d0700 1 --2- 172.21.15.58:0/1136520446 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6d080f5580 0x7f6d080f9d20 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:50.050 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:50.049+0000 7f6d0f7d0700 1 --2- 172.21.15.58:0/1136520446 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d080f6820 0x7f6d080f8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:50.050 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:50.049+0000 7f6d0f7d0700 1 -- 172.21.15.58:0/1136520446 >> 172.21.15.58:0/1136520446 conn(0x7f6d080f0ce0 msgr2=0x7f6d0805fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:50.050 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:50.049+0000 7f6d0f7d0700 1 -- 172.21.15.58:0/1136520446 shutdown_connections 2021-11-20T05:36:50.050 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:50.049+0000 7f6d0f7d0700 1 -- 172.21.15.58:0/1136520446 wait complete. 2021-11-20T05:36:50.051 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 62 2021-11-20T05:36:50.058 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:49.273931+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24367, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24367': {'gid': 24367, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 59, 'state': 'up:rejoin', 'state_seq': 15, 'addr': '172.21.15.187:6825/126010257', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 126010257}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 126010257}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:50.058 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24367:up:rejoin 2021-11-20T05:36:50.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:50 smithi058 conmon[33537]: cluster 2021-11-20T05:36:49.276858+0000 mon.smithi058 (mon. 2021-11-20T05:36:50.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:50 smithi058 conmon[33537]: 0) 691 : cluster [DBG] mds.? [v2:172.21.15.187:6824/126010257,v1:172.21.15.187:6825/126010257] up:rejoin 2021-11-20T05:36:50.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:50 smithi058 conmon[33537]: cluster 2021-11-20T05:36:49.276935+0000 mon.smithi058 (mon.0) 692 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:50.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:50 smithi058 conmon[33537]: audit 2021-11-20T05:36:50.048091+0000 mon.smithi058 (mon.0) 693 : audit [DBG] from='client.? 172.21.15.58:0/1136520446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:50.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:50 smithi187 conmon[36358]: cluster 2021-11-20T05:36:49.276858+0000 mon.smithi058 (mon.0) 691 : cluster [DBG] mds.? [v2:172.21.15.187:6824/126010257,v1:172.21.15.187:6825/126010257] up:rejoin 2021-11-20T05:36:50.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:50 smithi187 conmon[36358]: cluster 2021-11-20T05:36:49.276935+0000 mon.smithi058 (mon.0) 692 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:50.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:50 smithi187 conmon[36358]: audit 2021-11-20T05:36:50.048091+0000 mon.smithi058 (mon.0) 693 : audit [DBG] from='client.? 172.21.15.58:0/1136520446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:51.060 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:51.192 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.192+0000 7fedc8c41700 1 -- 172.21.15.58:0/2415265387 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fedc40f66b0 msgr2=0x7fedc40f6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:51.192 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.192+0000 7fedc8c41700 1 --2- 172.21.15.58:0/2415265387 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fedc40f66b0 0x7fedc40f6ad0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fedbc004660 tx=0x7fedbc009b30).stop 2021-11-20T05:36:51.193 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.192+0000 7fedc8c41700 1 -- 172.21.15.58:0/2415265387 shutdown_connections 2021-11-20T05:36:51.193 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.192+0000 7fedc8c41700 1 --2- 172.21.15.58:0/2415265387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedc40f7950 0x7fedc40f7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:36:51.193 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.192+0000 7fedc8c41700 1 --2- 172.21.15.58:0/2415265387 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fedc40f66b0 0x7fedc40f6ad0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:51.194 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.192+0000 7fedc8c41700 1 -- 172.21.15.58:0/2415265387 >> 172.21.15.58:0/2415265387 conn(0x7fedc40f1dd0 msgr2=0x7fedc40f4230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:51.194 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.192+0000 7fedc8c41700 1 -- 172.21.15.58:0/2415265387 shutdown_connections 2021-11-20T05:36:51.195 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.192+0000 7fedc8c41700 1 -- 172.21.15.58:0/2415265387 wait complete. 2021-11-20T05:36:51.195 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.193+0000 7fedc8c41700 1 Processor -- start 2021-11-20T05:36:51.196 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.193+0000 7fedc8c41700 1 -- start start 2021-11-20T05:36:51.196 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.193+0000 7fedc8c41700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedc40f66b0 0x7fedc4106f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:51.196 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.193+0000 7fedc8c41700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fedc40f7950 0x7fedc4107400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:51.197 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.193+0000 7fedc8c41700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc4107a40 con 0x7fedc40f7950 2021-11-20T05:36:51.197 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.193+0000 7fedc8c41700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc4107b80 con 0x7fedc40f66b0 2021-11-20T05:36:51.198 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.193+0000 7fedc1d9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fedc40f7950 0x7fedc4107400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:51.198 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.193+0000 7fedc1d9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fedc40f7950 0x7fedc4107400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41334/0 (socket says 172.21.15.58:41334) 2021-11-20T05:36:51.199 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.193+0000 7fedc1d9b700 1 -- 172.21.15.58:0/1854100353 learned_addr learned my addr 172.21.15.58:0/1854100353 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:51.199 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.193+0000 7fedc259c700 1 --2- 172.21.15.58:0/1854100353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedc40f66b0 0x7fedc4106f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:51.200 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.194+0000 7fedc1d9b700 1 -- 172.21.15.58:0/1854100353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedc40f66b0 msgr2=0x7fedc4106f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:51.200 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.194+0000 7fedc1d9b700 1 --2- 172.21.15.58:0/1854100353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedc40f66b0 0x7fedc4106f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:51.201 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.194+0000 7fedc1d9b700 1 -- 172.21.15.58:0/1854100353 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc005040 con 0x7fedc40f7950 2021-11-20T05:36:51.201 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.194+0000 7fedc1d9b700 1 --2- 172.21.15.58:0/1854100353 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fedc40f7950 0x7fedc4107400 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fedac00b410 tx=0x7fedac006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:51.202 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.194+0000 7fedbaffd700 1 -- 172.21.15.58:0/1854100353 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedac0125f0 con 0x7fedc40f7950 2021-11-20T05:36:51.202 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.194+0000 7fedbaffd700 1 -- 172.21.15.58:0/1854100353 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fedac00be80 con 0x7fedc40f7950 2021-11-20T05:36:51.202 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.194+0000 7fedbaffd700 1 -- 172.21.15.58:0/1854100353 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedac002ac0 con 0x7fedc40f7950 2021-11-20T05:36:51.203 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.194+0000 7fedc8c41700 1 -- 172.21.15.58:0/1854100353 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc410c620 con 0x7fedc40f7950 2021-11-20T05:36:51.203 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.194+0000 7fedc8c41700 1 -- 172.21.15.58:0/1854100353 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc410cbe0 con 0x7fedc40f7950 2021-11-20T05:36:51.204 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.195+0000 7fedc8c41700 1 -- 172.21.15.58:0/1854100353 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc4027a00 con 0x7fedc40f7950 2021-11-20T05:36:51.206 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.195+0000 7fedbaffd700 1 -- 172.21.15.58:0/1854100353 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fedac00f030 con 0x7fedc40f7950 2021-11-20T05:36:51.206 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.196+0000 7fedbaffd700 1 --2- 172.21.15.58:0/1854100353 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fedb005c2a0 0x7fedb005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:51.207 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.196+0000 7fedbaffd700 1 -- 172.21.15.58:0/1854100353 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fedac0765b0 con 0x7fedc40f7950 2021-11-20T05:36:51.207 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.196+0000 7fedc259c700 1 --2- 172.21.15.58:0/1854100353 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fedb005c2a0 0x7fedb005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:51.208 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.196+0000 7fedc259c700 1 --2- 172.21.15.58:0/1854100353 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fedb005c2a0 0x7fedb005e760 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fedbc004e30 tx=0x7fedbc009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:51.208 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.200+0000 7fedbaffd700 1 -- 172.21.15.58:0/1854100353 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fedac048f90 con 0x7fedc40f7950 2021-11-20T05:36:51.395 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.398+0000 7fedc8c41700 1 -- 172.21.15.58:0/1854100353 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedc40feec0 con 0x7fedc40f7950 2021-11-20T05:36:51.396 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.398+0000 7fedbaffd700 1 -- 172.21.15.58:0/1854100353 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2948 (secure 0 0 0) 0x7fedac04c5b0 con 0x7fedc40f7950 2021-11-20T05:36:51.396 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:51.396 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":63},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:49.273931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24367,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24367":{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":59,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:51.397 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.400+0000 7fedc8c41700 1 -- 172.21.15.58:0/1854100353 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fedb005c2a0 msgr2=0x7fedb005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:51.398 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.400+0000 7fedc8c41700 1 --2- 172.21.15.58:0/1854100353 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fedb005c2a0 0x7fedb005e760 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fedbc004e30 tx=0x7fedbc009b30).stop 2021-11-20T05:36:51.398 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.400+0000 7fedc8c41700 1 -- 172.21.15.58:0/1854100353 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fedc40f7950 msgr2=0x7fedc4107400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:51.398 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.400+0000 7fedc8c41700 1 --2- 172.21.15.58:0/1854100353 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fedc40f7950 0x7fedc4107400 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fedac00b410 tx=0x7fedac006ef0).stop 2021-11-20T05:36:51.399 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.400+0000 7fedc8c41700 1 -- 172.21.15.58:0/1854100353 shutdown_connections 2021-11-20T05:36:51.399 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.400+0000 7fedc8c41700 1 --2- 172.21.15.58:0/1854100353 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fedb005c2a0 0x7fedb005e760 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:51.399 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.400+0000 7fedc8c41700 1 --2- 172.21.15.58:0/1854100353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fedc40f66b0 0x7fedc4106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:51.399 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.400+0000 7fedc8c41700 1 --2- 172.21.15.58:0/1854100353 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fedc40f7950 0x7fedc4107400 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:51.400 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.400+0000 7fedc8c41700 1 -- 172.21.15.58:0/1854100353 >> 172.21.15.58:0/1854100353 conn(0x7fedc40f1dd0 msgr2=0x7fedc40fb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:51.400 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.400+0000 7fedc8c41700 1 -- 172.21.15.58:0/1854100353 shutdown_connections 2021-11-20T05:36:51.400 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:51.400+0000 7fedc8c41700 1 -- 172.21.15.58:0/1854100353 wait complete. 2021-11-20T05:36:51.400 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 63 2021-11-20T05:36:51.408 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:51 smithi058 conmon[33537]: cluster 2021-11-20T05:36:50.269906+0000 mgr.smithi058.zagdlk (mgr 2021-11-20T05:36:51.408 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:51 smithi058 conmon[33537]: .14164) 274 : cluster [DBG] pgmap v245: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:51.409 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:51 smithi058 conmon[33537]: cluster 2021-11-20T05:36:50.284088+0000 mon.smithi058 (mon.0) 694 : cluster [DBG] mds.? [v2:172.21.15.58:6826/3191333321,v1:172.21.15.58:6827/3191333321] up:standby 2021-11-20T05:36:51.409 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:51 smithi058 conmon[33537]: cluster 2021-11-20T05:36:50.284128+0000 mon.smithi058 2021-11-20T05:36:51.409 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:51 smithi058 conmon[33537]: (mon.0) 695 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:51.410 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:49.273931+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24367, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24367': {'gid': 24367, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 59, 'state': 'up:rejoin', 'state_seq': 15, 'addr': '172.21.15.187:6825/126010257', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 126010257}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 126010257}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:51.411 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24367:up:rejoin 2021-11-20T05:36:51.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:51 smithi187 conmon[36358]: cluster 2021-11-20T05:36:50.269906+0000 mgr.smithi058.zagdlk (mgr.14164) 274 : cluster [DBG] pgmap v245: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:51.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:51 smithi187 conmon[36358]: cluster 2021-11-20T05:36:50.284088+0000 mon.smithi058 (mon.0) 694 : cluster [DBG] mds.? [v2:172.21.15.58:6826/3191333321,v1:172.21.15.58:6827/3191333321] up:standby 2021-11-20T05:36:51.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:51 smithi187 conmon[36358]: cluster 2021-11-20 2021-11-20T05:36:51.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:51 smithi187 conmon[36358]: T05:36:50.284128+0000 mon.smithi058 (mon.0) 695 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:36:52.412 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:52.546 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.547+0000 7f0973e75700 1 -- 172.21.15.58:0/3824570625 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f096c0f6820 msgr2=0x7f096c0f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:52.547 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.547+0000 7f0973e75700 1 --2- 172.21.15.58:0/3824570625 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f096c0f6820 0x7f096c0f6c80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f0968004660 tx=0x7f0968009b30).stop 2021-11-20T05:36:52.547 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.547+0000 7f0973e75700 1 -- 172.21.15.58:0/3824570625 shutdown_connections 2021-11-20T05:36:52.547 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.547+0000 7f0973e75700 1 --2- 172.21.15.58:0/3824570625 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f096c0f6820 0x7f096c0f6c80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:52.547 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.547+0000 7f0973e75700 1 --2- 172.21.15.58:0/3824570625 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f096c0f5580 0x7f096c0f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:36:52.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.547+0000 7f0973e75700 1 -- 172.21.15.58:0/3824570625 >> 172.21.15.58:0/3824570625 conn(0x7f096c0f0ce0 msgr2=0x7f096c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:52.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.547+0000 7f0973e75700 1 -- 172.21.15.58:0/3824570625 shutdown_connections 2021-11-20T05:36:52.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.547+0000 7f0973e75700 1 -- 172.21.15.58:0/3824570625 wait complete. 2021-11-20T05:36:52.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.548+0000 7f0973e75700 1 Processor -- start 2021-11-20T05:36:52.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.548+0000 7f0973e75700 1 -- start start 2021-11-20T05:36:52.549 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.548+0000 7f0973e75700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f096c0f5580 0x7f096c105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:52.549 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.548+0000 7f0973e75700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f096c0f6820 0x7f096c106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:52.549 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.548+0000 7f0973e75700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c1068d0 con 0x7f096c0f6820 2021-11-20T05:36:52.549 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.548+0000 7f0973e75700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c106a10 con 0x7f096c0f5580 2021-11-20T05:36:52.549 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.548+0000 7f0971410700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f096c0f6820 0x7f096c106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:52.550 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.548+0000 7f0971410700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f096c0f6820 0x7f096c106290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41346/0 (socket says 172.21.15.58:41346) 2021-11-20T05:36:52.550 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.548+0000 7f0971410700 1 -- 172.21.15.58:0/2204831712 learned_addr learned my addr 172.21.15.58:0/2204831712 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:52.550 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.548+0000 7f0971c11700 1 --2- 172.21.15.58:0/2204831712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f096c0f5580 0x7f096c105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:52.550 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.548+0000 7f0971410700 1 -- 172.21.15.58:0/2204831712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f096c0f5580 msgr2=0x7f096c105da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:52.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.548+0000 7f0971410700 1 --2- 172.21.15.58:0/2204831712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f096c0f5580 0x7f096c105da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:52.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.548+0000 7f0971410700 1 -- 172.21.15.58:0/2204831712 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0968005040 con 0x7f096c0f6820 2021-11-20T05:36:52.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.548+0000 7f0971c11700 1 --2- 172.21.15.58:0/2204831712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f096c0f5580 0x7f096c105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:36:52.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.548+0000 7f0971410700 1 --2- 172.21.15.58:0/2204831712 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f096c0f6820 0x7f096c106290 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f0968007a40 tx=0x7f0968009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:52.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.549+0000 7f095e7fc700 1 -- 172.21.15.58:0/2204831712 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0968018070 con 0x7f096c0f6820 2021-11-20T05:36:52.552 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.549+0000 7f0973e75700 1 -- 172.21.15.58:0/2204831712 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096c10b450 con 0x7f096c0f6820 2021-11-20T05:36:52.552 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.549+0000 7f095e7fc700 1 -- 172.21.15.58:0/2204831712 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09680097d0 con 0x7f096c0f6820 2021-11-20T05:36:52.552 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.549+0000 7f095e7fc700 1 -- 172.21.15.58:0/2204831712 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f096801c620 con 0x7f096c0f6820 2021-11-20T05:36:52.552 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.549+0000 7f0973e75700 1 -- 172.21.15.58:0/2204831712 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f096c10b9b0 con 0x7f096c0f6820 2021-11-20T05:36:52.553 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.549+0000 7f0973e75700 1 -- 172.21.15.58:0/2204831712 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096c027a00 con 0x7f096c0f6820 2021-11-20T05:36:52.554 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.550+0000 7f095e7fc700 1 -- 172.21.15.58:0/2204831712 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f096801c890 con 0x7f096c0f6820 2021-11-20T05:36:52.554 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.550+0000 7f095e7fc700 1 --2- 172.21.15.58:0/2204831712 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f095805c2a0 0x7f095805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:52.554 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.550+0000 7f095e7fc700 1 -- 172.21.15.58:0/2204831712 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f0968078110 con 0x7f096c0f6820 2021-11-20T05:36:52.554 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.550+0000 7f0971c11700 1 --2- 172.21.15.58:0/2204831712 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f095805c2a0 0x7f095805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:52.555 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.551+0000 7f0971c11700 1 --2- 172.21.15.58:0/2204831712 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f095805c2a0 0x7f095805e760 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f096c0f6680 tx=0x7f0960008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:52.555 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.554+0000 7f095e7fc700 1 -- 172.21.15.58:0/2204831712 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f096804aad0 con 0x7f096c0f6820 2021-11-20T05:36:52.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:52 smithi187 conmon[36358]: audit 2021-11-20T05:36:51.399210+0000 mon.smithi058 (mon.0) 696 : audit [DBG] 2021-11-20T05:36:52.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:52 smithi187 conmon[36358]: from='client.? 172.21.15.58:0/1854100353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:52.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:52 smithi058 conmon[33537]: audit 2021-11-20T05:36:51.399210+0000 mon.smithi058 (mon 2021-11-20T05:36:52.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:52 smithi058 conmon[33537]: .0) 696 : audit [DBG] from='client.? 172.21.15.58:0/1854100353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:52.748 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.751+0000 7f0973e75700 1 -- 172.21.15.58:0/2204831712 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f096c0fddb0 con 0x7f096c0f6820 2021-11-20T05:36:52.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.752+0000 7f095e7fc700 1 -- 172.21.15.58:0/2204831712 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2948 (secure 0 0 0) 0x7f096804e0f0 con 0x7f096c0f6820 2021-11-20T05:36:52.750 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:52.750 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":63},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:49.273931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24367,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24367":{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":59,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:52.751 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.753+0000 7f0973e75700 1 -- 172.21.15.58:0/2204831712 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f095805c2a0 msgr2=0x7f095805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:52.751 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.753+0000 7f0973e75700 1 --2- 172.21.15.58:0/2204831712 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f095805c2a0 0x7f095805e760 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f096c0f6680 tx=0x7f0960008040).stop 2021-11-20T05:36:52.752 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.753+0000 7f0973e75700 1 -- 172.21.15.58:0/2204831712 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f096c0f6820 msgr2=0x7f096c106290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:52.752 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.753+0000 7f0973e75700 1 --2- 172.21.15.58:0/2204831712 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f096c0f6820 0x7f096c106290 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f0968007a40 tx=0x7f0968009b30).stop 2021-11-20T05:36:52.752 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.753+0000 7f0973e75700 1 -- 172.21.15.58:0/2204831712 shutdown_connections 2021-11-20T05:36:52.752 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.753+0000 7f0973e75700 1 --2- 172.21.15.58:0/2204831712 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f095805c2a0 0x7f095805e760 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:52.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.753+0000 7f0973e75700 1 --2- 172.21.15.58:0/2204831712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f096c0f5580 0x7f096c105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:52.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.753+0000 7f0973e75700 1 --2- 172.21.15.58:0/2204831712 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f096c0f6820 0x7f096c106290 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:52.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.753+0000 7f0973e75700 1 -- 172.21.15.58:0/2204831712 >> 172.21.15.58:0/2204831712 conn(0x7f096c0f0ce0 msgr2=0x7f096c0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:52.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.753+0000 7f0973e75700 1 -- 172.21.15.58:0/2204831712 shutdown_connections 2021-11-20T05:36:52.754 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:52.753+0000 7f0973e75700 1 -- 172.21.15.58:0/2204831712 wait complete. 2021-11-20T05:36:52.754 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 63 2021-11-20T05:36:52.762 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:49.273931+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24367, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24367': {'gid': 24367, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 59, 'state': 'up:rejoin', 'state_seq': 15, 'addr': '172.21.15.187:6825/126010257', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 126010257}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 126010257}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:52.763 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24367:up:rejoin 2021-11-20T05:36:53.575 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:53 smithi187 conmon[36358]: cluster 2021-11-20T05:36:52.270331+0000 mgr.smithi058.zagdlk (mgr.14164) 275 : cluster 2021-11-20T05:36:53.576 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:53 smithi187 conmon[36358]: [DBG] pgmap v246: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:53.576 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:53 smithi187 conmon[36358]: audit 2021-11-20T05:36:52.752791+0000 mon.smithi058 (mon.0) 697 : audit [DBG] from='client.? 172.21.15.58:0/2204831712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:53.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:53 smithi058 conmon[33537]: cluster 2021-11-20T05:36:52.270331+0000 mgr.smithi058.zagdlk (mgr.14164) 275 2021-11-20T05:36:53.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:53 smithi058 conmon[33537]: : cluster [DBG] pgmap v246: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:53.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:53 smithi058 conmon[33537]: audit 2021-11-20T05:36:52.752791+0000 mon.smithi058 (mon.0) 697 : audit [DBG] 2021-11-20T05:36:53.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:53 smithi058 conmon[33537]: from='client.? 172.21.15.58:0/2204831712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:53.764 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:53.900 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.900+0000 7fde41239700 1 -- 172.21.15.58:0/3160829606 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fde3c0f66c0 msgr2=0x7fde3c0f6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:53.900 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.900+0000 7fde41239700 1 --2- 172.21.15.58:0/3160829606 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fde3c0f66c0 0x7fde3c0f6ae0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fde24004660 tx=0x7fde24009b30).stop 2021-11-20T05:36:53.900 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.900+0000 7fde41239700 1 -- 172.21.15.58:0/3160829606 shutdown_connections 2021-11-20T05:36:53.901 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.900+0000 7fde41239700 1 --2- 172.21.15.58:0/3160829606 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde3c0f7960 0x7fde3c0f7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:36:53.901 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.900+0000 7fde41239700 1 --2- 172.21.15.58:0/3160829606 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fde3c0f66c0 0x7fde3c0f6ae0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:53.901 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.900+0000 7fde41239700 1 -- 172.21.15.58:0/3160829606 >> 172.21.15.58:0/3160829606 conn(0x7fde3c0f1e40 msgr2=0x7fde3c0f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:53.901 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.900+0000 7fde41239700 1 -- 172.21.15.58:0/3160829606 shutdown_connections 2021-11-20T05:36:53.901 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.900+0000 7fde41239700 1 -- 172.21.15.58:0/3160829606 wait complete. 2021-11-20T05:36:53.902 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.900+0000 7fde41239700 1 Processor -- start 2021-11-20T05:36:53.902 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.900+0000 7fde41239700 1 -- start start 2021-11-20T05:36:53.902 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.901+0000 7fde41239700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde3c0f66c0 0x7fde3c106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:53.902 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.901+0000 7fde41239700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fde3c0f7960 0x7fde3c1073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:53.903 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.901+0000 7fde41239700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3c1079f0 con 0x7fde3c0f7960 2021-11-20T05:36:53.903 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.901+0000 7fde41239700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3c107b30 con 0x7fde3c0f66c0 2021-11-20T05:36:53.903 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.901+0000 7fde3a59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fde3c0f7960 0x7fde3c1073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:53.903 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.901+0000 7fde3a59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fde3c0f7960 0x7fde3c1073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41356/0 (socket says 172.21.15.58:41356) 2021-11-20T05:36:53.904 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.901+0000 7fde3a59c700 1 -- 172.21.15.58:0/1718818754 learned_addr learned my addr 172.21.15.58:0/1718818754 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:53.904 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.901+0000 7fde3ad9d700 1 --2- 172.21.15.58:0/1718818754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde3c0f66c0 0x7fde3c106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:53.904 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.901+0000 7fde3a59c700 1 -- 172.21.15.58:0/1718818754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde3c0f66c0 msgr2=0x7fde3c106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:53.904 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.901+0000 7fde3a59c700 1 --2- 172.21.15.58:0/1718818754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde3c0f66c0 0x7fde3c106ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:53.904 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.901+0000 7fde3a59c700 1 -- 172.21.15.58:0/1718818754 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde24005040 con 0x7fde3c0f7960 2021-11-20T05:36:53.905 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.901+0000 7fde3ad9d700 1 --2- 172.21.15.58:0/1718818754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde3c0f66c0 0x7fde3c106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:36:53.905 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.901+0000 7fde3a59c700 1 --2- 172.21.15.58:0/1718818754 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fde3c0f7960 0x7fde3c1073b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fde2c00b410 tx=0x7fde2c006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:53.905 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.901+0000 7fde337fe700 1 -- 172.21.15.58:0/1718818754 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde2c0125f0 con 0x7fde3c0f7960 2021-11-20T05:36:53.905 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.902+0000 7fde41239700 1 -- 172.21.15.58:0/1718818754 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde3c10c5d0 con 0x7fde3c0f7960 2021-11-20T05:36:53.906 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.902+0000 7fde337fe700 1 -- 172.21.15.58:0/1718818754 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde2c00be80 con 0x7fde3c0f7960 2021-11-20T05:36:53.906 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.902+0000 7fde337fe700 1 -- 172.21.15.58:0/1718818754 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde2c002ac0 con 0x7fde3c0f7960 2021-11-20T05:36:53.906 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.902+0000 7fde41239700 1 -- 172.21.15.58:0/1718818754 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde3c10cb90 con 0x7fde3c0f7960 2021-11-20T05:36:53.906 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.902+0000 7fde41239700 1 -- 172.21.15.58:0/1718818754 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde3c027a00 con 0x7fde3c0f7960 2021-11-20T05:36:53.907 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.903+0000 7fde337fe700 1 -- 172.21.15.58:0/1718818754 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fde2c00f030 con 0x7fde3c0f7960 2021-11-20T05:36:53.908 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.903+0000 7fde337fe700 1 --2- 172.21.15.58:0/1718818754 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fde2805c2a0 0x7fde2805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:53.908 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.903+0000 7fde337fe700 1 -- 172.21.15.58:0/1718818754 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fde2c076660 con 0x7fde3c0f7960 2021-11-20T05:36:53.908 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.903+0000 7fde3ad9d700 1 --2- 172.21.15.58:0/1718818754 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fde2805c2a0 0x7fde2805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:53.908 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.904+0000 7fde3ad9d700 1 --2- 172.21.15.58:0/1718818754 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fde2805c2a0 0x7fde2805e760 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fde24004e30 tx=0x7fde24009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:53.909 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:53.907+0000 7fde337fe700 1 -- 172.21.15.58:0/1718818754 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde2c048fc0 con 0x7fde3c0f7960 2021-11-20T05:36:54.101 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:54.104+0000 7fde41239700 1 -- 172.21.15.58:0/1718818754 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde3c0feed0 con 0x7fde3c0f7960 2021-11-20T05:36:54.102 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:54.105+0000 7fde337fe700 1 -- 172.21.15.58:0/1718818754 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2948 (secure 0 0 0) 0x7fde2c04c5e0 con 0x7fde3c0f7960 2021-11-20T05:36:54.102 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:54.102 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":63},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:49.273931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24367,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24367":{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":59,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:54.104 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:54.106+0000 7fde41239700 1 -- 172.21.15.58:0/1718818754 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fde2805c2a0 msgr2=0x7fde2805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:54.104 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:54.106+0000 7fde41239700 1 --2- 172.21.15.58:0/1718818754 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fde2805c2a0 0x7fde2805e760 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fde24004e30 tx=0x7fde24009b30).stop 2021-11-20T05:36:54.104 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:54.106+0000 7fde41239700 1 -- 172.21.15.58:0/1718818754 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fde3c0f7960 msgr2=0x7fde3c1073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:54.104 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:54.106+0000 7fde41239700 1 --2- 172.21.15.58:0/1718818754 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fde3c0f7960 0x7fde3c1073b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fde2c00b410 tx=0x7fde2c006ef0).stop 2021-11-20T05:36:54.105 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:54.106+0000 7fde41239700 1 -- 172.21.15.58:0/1718818754 shutdown_connections 2021-11-20T05:36:54.105 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:54.106+0000 7fde41239700 1 --2- 172.21.15.58:0/1718818754 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fde2805c2a0 0x7fde2805e760 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:54.105 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:54.106+0000 7fde41239700 1 --2- 172.21.15.58:0/1718818754 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde3c0f66c0 0x7fde3c106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:54.105 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:54.106+0000 7fde41239700 1 --2- 172.21.15.58:0/1718818754 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fde3c0f7960 0x7fde3c1073b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:54.106 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:54.106+0000 7fde41239700 1 -- 172.21.15.58:0/1718818754 >> 172.21.15.58:0/1718818754 conn(0x7fde3c0f1e40 msgr2=0x7fde3c0fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:54.106 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:54.106+0000 7fde41239700 1 -- 172.21.15.58:0/1718818754 shutdown_connections 2021-11-20T05:36:54.106 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:54.107+0000 7fde41239700 1 -- 172.21.15.58:0/1718818754 wait complete. 2021-11-20T05:36:54.106 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 63 2021-11-20T05:36:54.115 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:49.273931+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24367, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24367': {'gid': 24367, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 59, 'state': 'up:rejoin', 'state_seq': 15, 'addr': '172.21.15.187:6825/126010257', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 126010257}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 126010257}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:54.116 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24367:up:rejoin 2021-11-20T05:36:54.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:54 smithi058 conmon[33537]: audit 2021-11-20T05:36:54.105620+0000 mon.smithi058 (mon.0) 698 : audit [DBG] from='client.? 172.21.15.58:0/1718818754' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:54.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:54 smithi187 conmon[36358]: audit 2021-11-20T05:36:54.105620+0000 mon.smithi058 (mon.0) 698 2021-11-20T05:36:54.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:54 smithi187 conmon[36358]: : audit [DBG] from='client.? 172.21.15.58:0/1718818754' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:55.118 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:55.252 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.252+0000 7f92cbbc3700 1 -- 172.21.15.58:0/102365888 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f92c40fbb50 msgr2=0x7f92c40fdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:55.252 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.252+0000 7f92cbbc3700 1 --2- 172.21.15.58:0/102365888 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f92c40fbb50 0x7f92c40fdf40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f92b8004660 tx=0x7f92b8009b30).stop 2021-11-20T05:36:55.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.252+0000 7f92cbbc3700 1 -- 172.21.15.58:0/102365888 shutdown_connections 2021-11-20T05:36:55.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.252+0000 7f92cbbc3700 1 --2- 172.21.15.58:0/102365888 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f92c40fbb50 0x7f92c40fdf40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:55.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.252+0000 7f92cbbc3700 1 --2- 172.21.15.58:0/102365888 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92c4060730 0x7f92c40fb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:55.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.252+0000 7f92cbbc3700 1 -- 172.21.15.58:0/102365888 >> 172.21.15.58:0/102365888 conn(0x7f92c40f0ce0 msgr2=0x7f92c40f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:55.254 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.252+0000 7f92cbbc3700 1 -- 172.21.15.58:0/102365888 shutdown_connections 2021-11-20T05:36:55.254 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.252+0000 7f92cbbc3700 1 -- 172.21.15.58:0/102365888 wait complete. 2021-11-20T05:36:55.254 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.253+0000 7f92cbbc3700 1 Processor -- start 2021-11-20T05:36:55.254 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.253+0000 7f92cbbc3700 1 -- start start 2021-11-20T05:36:55.255 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.253+0000 7f92cbbc3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92c4060730 0x7f92c41090a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:55.255 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.253+0000 7f92cbbc3700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f92c40fbb50 0x7f92c4109590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:55.255 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.253+0000 7f92cbbc3700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c4109b40 con 0x7f92c40fbb50 2021-11-20T05:36:55.255 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.253+0000 7f92cbbc3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c4109c80 con 0x7f92c4060730 2021-11-20T05:36:55.255 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.253+0000 7f92c915e700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f92c40fbb50 0x7f92c4109590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:55.256 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.253+0000 7f92c915e700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f92c40fbb50 0x7f92c4109590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41368/0 (socket says 172.21.15.58:41368) 2021-11-20T05:36:55.256 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.253+0000 7f92c915e700 1 -- 172.21.15.58:0/2160355835 learned_addr learned my addr 172.21.15.58:0/2160355835 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:55.256 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.253+0000 7f92c995f700 1 --2- 172.21.15.58:0/2160355835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92c4060730 0x7f92c41090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:55.256 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.254+0000 7f92c915e700 1 -- 172.21.15.58:0/2160355835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92c4060730 msgr2=0x7f92c41090a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:55.257 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.254+0000 7f92c915e700 1 --2- 172.21.15.58:0/2160355835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92c4060730 0x7f92c41090a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:55.257 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.254+0000 7f92c915e700 1 -- 172.21.15.58:0/2160355835 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b8005040 con 0x7f92c40fbb50 2021-11-20T05:36:55.257 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.254+0000 7f92c915e700 1 --2- 172.21.15.58:0/2160355835 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f92c40fbb50 0x7f92c4109590 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f92b8004e30 tx=0x7f92b8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:55.259 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.255+0000 7f92c27fc700 1 -- 172.21.15.58:0/2160355835 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92b8018070 con 0x7f92c40fbb50 2021-11-20T05:36:55.259 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.255+0000 7f92c27fc700 1 -- 172.21.15.58:0/2160355835 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f92b8013410 con 0x7f92c40fbb50 2021-11-20T05:36:55.260 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.255+0000 7f92c27fc700 1 -- 172.21.15.58:0/2160355835 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92b801c3c0 con 0x7f92c40fbb50 2021-11-20T05:36:55.260 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.255+0000 7f92cbbc3700 1 -- 172.21.15.58:0/2160355835 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c410e6c0 con 0x7f92c40fbb50 2021-11-20T05:36:55.260 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.256+0000 7f92cbbc3700 1 -- 172.21.15.58:0/2160355835 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c410ec80 con 0x7f92c40fbb50 2021-11-20T05:36:55.260 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.257+0000 7f92cbbc3700 1 -- 172.21.15.58:0/2160355835 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c4027a00 con 0x7f92c40fbb50 2021-11-20T05:36:55.261 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.257+0000 7f92c27fc700 1 -- 172.21.15.58:0/2160355835 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f92b801d420 con 0x7f92c40fbb50 2021-11-20T05:36:55.261 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.257+0000 7f92c27fc700 1 --2- 172.21.15.58:0/2160355835 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f92ac05c2a0 0x7f92ac05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:55.261 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.257+0000 7f92c27fc700 1 -- 172.21.15.58:0/2160355835 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f92b8077e10 con 0x7f92c40fbb50 2021-11-20T05:36:55.261 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.257+0000 7f92c995f700 1 --2- 172.21.15.58:0/2160355835 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f92ac05c2a0 0x7f92ac05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:55.262 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.258+0000 7f92c995f700 1 --2- 172.21.15.58:0/2160355835 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f92ac05c2a0 0x7f92ac05e760 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f92bc0096c0 tx=0x7f92bc009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:55.262 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.261+0000 7f92c27fc700 1 -- 172.21.15.58:0/2160355835 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92b804a850 con 0x7f92c40fbb50 2021-11-20T05:36:55.461 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.463+0000 7f92cbbc3700 1 -- 172.21.15.58:0/2160355835 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92c4101020 con 0x7f92c40fbb50 2021-11-20T05:36:55.461 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.463+0000 7f92c27fc700 1 -- 172.21.15.58:0/2160355835 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2948 (secure 0 0 0) 0x7f92b804de70 con 0x7f92c40fbb50 2021-11-20T05:36:55.462 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:55.462 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":63},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:49.273931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24367,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24367":{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":59,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:55.463 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.465+0000 7f92cbbc3700 1 -- 172.21.15.58:0/2160355835 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f92ac05c2a0 msgr2=0x7f92ac05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:55.464 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.465+0000 7f92cbbc3700 1 --2- 172.21.15.58:0/2160355835 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f92ac05c2a0 0x7f92ac05e760 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f92bc0096c0 tx=0x7f92bc009be0).stop 2021-11-20T05:36:55.464 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.465+0000 7f92cbbc3700 1 -- 172.21.15.58:0/2160355835 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f92c40fbb50 msgr2=0x7f92c4109590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:55.465 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.465+0000 7f92cbbc3700 1 --2- 172.21.15.58:0/2160355835 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f92c40fbb50 0x7f92c4109590 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f92b8004e30 tx=0x7f92b8009b30).stop 2021-11-20T05:36:55.465 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.465+0000 7f92cbbc3700 1 -- 172.21.15.58:0/2160355835 shutdown_connections 2021-11-20T05:36:55.465 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.465+0000 7f92cbbc3700 1 --2- 172.21.15.58:0/2160355835 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f92ac05c2a0 0x7f92ac05e760 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:55.466 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.465+0000 7f92cbbc3700 1 --2- 172.21.15.58:0/2160355835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92c4060730 0x7f92c41090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:55.466 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.465+0000 7f92cbbc3700 1 --2- 172.21.15.58:0/2160355835 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f92c40fbb50 0x7f92c4109590 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:55.466 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.465+0000 7f92cbbc3700 1 -- 172.21.15.58:0/2160355835 >> 172.21.15.58:0/2160355835 conn(0x7f92c40f0ce0 msgr2=0x7f92c40f30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:55.466 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.465+0000 7f92cbbc3700 1 -- 172.21.15.58:0/2160355835 shutdown_connections 2021-11-20T05:36:55.466 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:55.465+0000 7f92cbbc3700 1 -- 172.21.15.58:0/2160355835 wait complete. 2021-11-20T05:36:55.467 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 63 2021-11-20T05:36:55.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:55 smithi058 conmon[33537]: cluster 2021-11-20T05:36:54.270770+0000 mgr.smithi058.zagdlk (mgr.14164) 276 : cluster [DBG] pgmap v247: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:55.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:49.273931+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24367, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24367': {'gid': 24367, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 59, 'state': 'up:rejoin', 'state_seq': 15, 'addr': '172.21.15.187:6825/126010257', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 126010257}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 126010257}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:55.475 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24367:up:rejoin 2021-11-20T05:36:55.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:55 smithi187 conmon[36358]: cluster 2021-11-20T05:36:54.270770+0000 mgr.smithi058.zagdlk (mgr.14164) 276 : cluster [DBG] pgmap v247: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:56.476 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:56.609 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.611+0000 7f8faa82f700 1 -- 172.21.15.58:0/2526540941 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8fa4060690 msgr2=0x7f8fa40f92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:56.610 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.611+0000 7f8faa82f700 1 --2- 172.21.15.58:0/2526540941 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8fa4060690 0x7f8fa40f92e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f8f94004660 tx=0x7f8f94009b30).stop 2021-11-20T05:36:56.610 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.611+0000 7f8faa82f700 1 -- 172.21.15.58:0/2526540941 shutdown_connections 2021-11-20T05:36:56.610 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.611+0000 7f8faa82f700 1 --2- 172.21.15.58:0/2526540941 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8fa40f9860 0x7f8fa40fbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:36:56.611 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.611+0000 7f8faa82f700 1 --2- 172.21.15.58:0/2526540941 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8fa4060690 0x7f8fa40f92e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:56.611 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.611+0000 7f8faa82f700 1 -- 172.21.15.58:0/2526540941 >> 172.21.15.58:0/2526540941 conn(0x7f8fa40f0ce0 msgr2=0x7f8fa40f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:56.611 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.611+0000 7f8faa82f700 1 -- 172.21.15.58:0/2526540941 shutdown_connections 2021-11-20T05:36:56.611 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.611+0000 7f8faa82f700 1 -- 172.21.15.58:0/2526540941 wait complete. 2021-11-20T05:36:56.612 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.612+0000 7f8faa82f700 1 Processor -- start 2021-11-20T05:36:56.614 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.612+0000 7f8faa82f700 1 -- start start 2021-11-20T05:36:56.615 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.612+0000 7f8faa82f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8fa4060690 0x7f8fa4106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:56.615 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.612+0000 7f8faa82f700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8fa40f9860 0x7f8fa41073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:56.615 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.612+0000 7f8faa82f700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa41079e0 con 0x7f8fa40f9860 2021-11-20T05:36:56.615 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.612+0000 7f8faa82f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa4107b20 con 0x7f8fa4060690 2021-11-20T05:36:56.616 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.612+0000 7f8fa37fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8fa40f9860 0x7f8fa41073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:56.616 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.613+0000 7f8fa37fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8fa40f9860 0x7f8fa41073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41380/0 (socket says 172.21.15.58:41380) 2021-11-20T05:36:56.616 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.613+0000 7f8fa37fe700 1 -- 172.21.15.58:0/2688728536 learned_addr learned my addr 172.21.15.58:0/2688728536 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:56.616 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.613+0000 7f8fa3fff700 1 --2- 172.21.15.58:0/2688728536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8fa4060690 0x7f8fa4106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:56.617 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.613+0000 7f8fa37fe700 1 -- 172.21.15.58:0/2688728536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8fa4060690 msgr2=0x7f8fa4106eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:56.617 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.613+0000 7f8fa37fe700 1 --2- 172.21.15.58:0/2688728536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8fa4060690 0x7f8fa4106eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:56.617 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.613+0000 7f8fa37fe700 1 -- 172.21.15.58:0/2688728536 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f94005040 con 0x7f8fa40f9860 2021-11-20T05:36:56.618 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.613+0000 7f8fa37fe700 1 --2- 172.21.15.58:0/2688728536 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8fa40f9860 0x7f8fa41073a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f8f980093f0 tx=0x7f8f98006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:56.618 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.613+0000 7f8fa0ff9700 1 -- 172.21.15.58:0/2688728536 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f98013640 con 0x7f8fa40f9860 2021-11-20T05:36:56.618 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.613+0000 7f8fa0ff9700 1 -- 172.21.15.58:0/2688728536 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f98009e40 con 0x7f8fa40f9860 2021-11-20T05:36:56.619 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.613+0000 7f8fa0ff9700 1 -- 172.21.15.58:0/2688728536 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f9800dbf0 con 0x7f8fa40f9860 2021-11-20T05:36:56.619 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.613+0000 7f8faa82f700 1 -- 172.21.15.58:0/2688728536 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa410c5c0 con 0x7f8fa40f9860 2021-11-20T05:36:56.619 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.613+0000 7f8faa82f700 1 -- 172.21.15.58:0/2688728536 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa410cb80 con 0x7f8fa40f9860 2021-11-20T05:36:56.619 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.614+0000 7f8faa82f700 1 -- 172.21.15.58:0/2688728536 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa4027a00 con 0x7f8fa40f9860 2021-11-20T05:36:56.620 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.615+0000 7f8fa0ff9700 1 -- 172.21.15.58:0/2688728536 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8f9801a060 con 0x7f8fa40f9860 2021-11-20T05:36:56.620 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.615+0000 7f8fa0ff9700 1 --2- 172.21.15.58:0/2688728536 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8f8c05c2a0 0x7f8f8c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:56.621 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.615+0000 7f8fa0ff9700 1 -- 172.21.15.58:0/2688728536 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f8f98076210 con 0x7f8fa40f9860 2021-11-20T05:36:56.621 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.615+0000 7f8fa3fff700 1 --2- 172.21.15.58:0/2688728536 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8f8c05c2a0 0x7f8f8c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:56.621 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.615+0000 7f8fa3fff700 1 --2- 172.21.15.58:0/2688728536 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8f8c05c2a0 0x7f8f8c05e760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f8f94013580 tx=0x7f8f94009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:56.622 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.619+0000 7f8fa0ff9700 1 -- 172.21.15.58:0/2688728536 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f98048b70 con 0x7f8fa40f9860 2021-11-20T05:36:56.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:56 smithi187 conmon[36358]: audit 2021-11-20T05:36:55.465228+0000 mon.smithi058 (mon.0) 699 : audit [DBG] from='client.? 172.21.15.58:0/2160355835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:56.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:56 smithi058 conmon[33537]: audit 2021-11-20T05:36:55.465228+0000 mon.smithi058 ( 2021-11-20T05:36:56.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:56 smithi058 conmon[33537]: mon.0) 699 : audit [DBG] from='client.? 172.21.15.58:0/2160355835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:56.812 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.814+0000 7f8faa82f700 1 -- 172.21.15.58:0/2688728536 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fa40feeb0 con 0x7f8fa40f9860 2021-11-20T05:36:56.812 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.814+0000 7f8fa0ff9700 1 -- 172.21.15.58:0/2688728536 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2948 (secure 0 0 0) 0x7f8f9804c190 con 0x7f8fa40f9860 2021-11-20T05:36:56.813 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:56.813 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":63},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:49.273931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24367,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24367":{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":59,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:56.815 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.816+0000 7f8faa82f700 1 -- 172.21.15.58:0/2688728536 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8f8c05c2a0 msgr2=0x7f8f8c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:56.815 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.816+0000 7f8faa82f700 1 --2- 172.21.15.58:0/2688728536 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8f8c05c2a0 0x7f8f8c05e760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f8f94013580 tx=0x7f8f94009b30).stop 2021-11-20T05:36:56.815 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.816+0000 7f8faa82f700 1 -- 172.21.15.58:0/2688728536 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8fa40f9860 msgr2=0x7f8fa41073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:56.815 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.816+0000 7f8faa82f700 1 --2- 172.21.15.58:0/2688728536 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8fa40f9860 0x7f8fa41073a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f8f980093f0 tx=0x7f8f98006ec0).stop 2021-11-20T05:36:56.816 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.816+0000 7f8faa82f700 1 -- 172.21.15.58:0/2688728536 shutdown_connections 2021-11-20T05:36:56.816 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.816+0000 7f8faa82f700 1 --2- 172.21.15.58:0/2688728536 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8f8c05c2a0 0x7f8f8c05e760 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:56.816 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.816+0000 7f8faa82f700 1 --2- 172.21.15.58:0/2688728536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8fa4060690 0x7f8fa4106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:56.816 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.816+0000 7f8faa82f700 1 --2- 172.21.15.58:0/2688728536 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8fa40f9860 0x7f8fa41073a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:56.817 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.816+0000 7f8faa82f700 1 -- 172.21.15.58:0/2688728536 >> 172.21.15.58:0/2688728536 conn(0x7f8fa40f0ce0 msgr2=0x7f8fa40f30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:56.817 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.816+0000 7f8faa82f700 1 -- 172.21.15.58:0/2688728536 shutdown_connections 2021-11-20T05:36:56.817 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:56.816+0000 7f8faa82f700 1 -- 172.21.15.58:0/2688728536 wait complete. 2021-11-20T05:36:56.817 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 63 2021-11-20T05:36:56.826 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:49.273931+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24367, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24367': {'gid': 24367, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 59, 'state': 'up:rejoin', 'state_seq': 15, 'addr': '172.21.15.187:6825/126010257', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 126010257}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 126010257}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:56.826 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24367:up:rejoin 2021-11-20T05:36:57.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:57 smithi187 conmon[36358]: cluster 2021-11-20T05:36:56.271266+0000 mgr.smithi058.zagdlk (mgr.14164) 277 2021-11-20T05:36:57.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:57 smithi187 conmon[36358]: : cluster [DBG] pgmap v248: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:57.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:57 smithi187 conmon[36358]: audit 2021-11-20T05:36:56.816312+0000 mon.smithi058 (mon.0) 700 : audit [DBG] from='client.? 172.21.15.58:0/2688728536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:57.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:57 smithi058 conmon[33537]: cluster 2021-11-20T05:36:56.271266+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:36:57.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:57 smithi058 conmon[33537]: ) 277 : cluster [DBG] pgmap v248: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:36:57.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:57 smithi058 conmon[33537]: audit 2021-11-20T05:36:56.816312+0000 mon.smithi058 (mon.0) 700 : audit [DBG] from='client.? 172.21.15.58:0/2688728536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:57.827 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:57.962 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.962+0000 7fac74fa7700 1 -- 172.21.15.58:0/3149168944 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fac700f6810 msgr2=0x7fac700f6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:57.962 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.962+0000 7fac74fa7700 1 --2- 172.21.15.58:0/3149168944 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fac700f6810 0x7fac700f6c70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fac64004660 tx=0x7fac64009b30).stop 2021-11-20T05:36:57.963 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.962+0000 7fac74fa7700 1 -- 172.21.15.58:0/3149168944 shutdown_connections 2021-11-20T05:36:57.963 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.962+0000 7fac74fa7700 1 --2- 172.21.15.58:0/3149168944 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fac700f6810 0x7fac700f6c70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:57.963 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.962+0000 7fac74fa7700 1 --2- 172.21.15.58:0/3149168944 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac700f5570 0x7fac700f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:57.963 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.962+0000 7fac74fa7700 1 -- 172.21.15.58:0/3149168944 >> 172.21.15.58:0/3149168944 conn(0x7fac700f0cf0 msgr2=0x7fac700f3110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:57.964 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.962+0000 7fac74fa7700 1 -- 172.21.15.58:0/3149168944 shutdown_connections 2021-11-20T05:36:57.964 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.962+0000 7fac74fa7700 1 -- 172.21.15.58:0/3149168944 wait complete. 2021-11-20T05:36:57.964 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.962+0000 7fac74fa7700 1 Processor -- start 2021-11-20T05:36:57.964 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.962+0000 7fac74fa7700 1 -- start start 2021-11-20T05:36:57.965 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.962+0000 7fac74fa7700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fac700f5570 0x7fac700604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:57.965 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.962+0000 7fac74fa7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac700f6810 0x7fac7010e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:57.965 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.962+0000 7fac74fa7700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac70107a60 con 0x7fac700f5570 2021-11-20T05:36:57.965 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.962+0000 7fac74fa7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7010e570 con 0x7fac700f6810 2021-11-20T05:36:57.965 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.963+0000 7fac6e59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fac700f5570 0x7fac700604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:57.966 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.963+0000 7fac6e59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fac700f5570 0x7fac700604f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41388/0 (socket says 172.21.15.58:41388) 2021-11-20T05:36:57.966 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.963+0000 7fac6e59c700 1 -- 172.21.15.58:0/3108632218 learned_addr learned my addr 172.21.15.58:0/3108632218 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:57.966 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.963+0000 7fac6dd9b700 1 --2- 172.21.15.58:0/3108632218 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac700f6810 0x7fac7010e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:57.966 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.963+0000 7fac6e59c700 1 -- 172.21.15.58:0/3108632218 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac700f6810 msgr2=0x7fac7010e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:57.966 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.963+0000 7fac6e59c700 1 --2- 172.21.15.58:0/3108632218 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac700f6810 0x7fac7010e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:57.967 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.963+0000 7fac6e59c700 1 -- 172.21.15.58:0/3108632218 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac64005040 con 0x7fac700f5570 2021-11-20T05:36:57.967 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.963+0000 7fac6dd9b700 1 --2- 172.21.15.58:0/3108632218 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac700f6810 0x7fac7010e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:36:57.967 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.963+0000 7fac6e59c700 1 --2- 172.21.15.58:0/3108632218 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fac700f5570 0x7fac700604f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fac60002700 tx=0x7fac60009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:57.968 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.963+0000 7fac5effd700 1 -- 172.21.15.58:0/3108632218 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac60009320 con 0x7fac700f5570 2021-11-20T05:36:57.968 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.963+0000 7fac5effd700 1 -- 172.21.15.58:0/3108632218 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac6000f040 con 0x7fac700f5570 2021-11-20T05:36:57.968 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.963+0000 7fac5effd700 1 -- 172.21.15.58:0/3108632218 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac60014900 con 0x7fac700f5570 2021-11-20T05:36:57.968 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.963+0000 7fac74fa7700 1 -- 172.21.15.58:0/3108632218 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac7010e710 con 0x7fac700f5570 2021-11-20T05:36:57.969 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.963+0000 7fac74fa7700 1 -- 172.21.15.58:0/3108632218 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac7010eca0 con 0x7fac700f5570 2021-11-20T05:36:57.969 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.964+0000 7fac74fa7700 1 -- 172.21.15.58:0/3108632218 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac70046330 con 0x7fac700f5570 2021-11-20T05:36:57.969 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.964+0000 7fac5effd700 1 -- 172.21.15.58:0/3108632218 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fac60009480 con 0x7fac700f5570 2021-11-20T05:36:57.970 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.965+0000 7fac5effd700 1 --2- 172.21.15.58:0/3108632218 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fac5805c2a0 0x7fac5805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:57.970 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.965+0000 7fac5effd700 1 -- 172.21.15.58:0/3108632218 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fac60076420 con 0x7fac700f5570 2021-11-20T05:36:57.971 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.965+0000 7fac6dd9b700 1 --2- 172.21.15.58:0/3108632218 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fac5805c2a0 0x7fac5805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:57.971 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.965+0000 7fac6dd9b700 1 --2- 172.21.15.58:0/3108632218 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fac5805c2a0 0x7fac5805e760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fac64009d60 tx=0x7fac64009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:57.971 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:57.969+0000 7fac5effd700 1 -- 172.21.15.58:0/3108632218 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac60048e60 con 0x7fac700f5570 2021-11-20T05:36:58.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:58.167+0000 7fac74fa7700 1 -- 172.21.15.58:0/3108632218 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac700fdd00 con 0x7fac700f5570 2021-11-20T05:36:58.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:58.167+0000 7fac5effd700 1 -- 172.21.15.58:0/3108632218 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2948 (secure 0 0 0) 0x7fac6004c480 con 0x7fac700f5570 2021-11-20T05:36:58.166 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:58.167 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":63},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:49.273931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24367,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24367":{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":59,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:58.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:58.169+0000 7fac74fa7700 1 -- 172.21.15.58:0/3108632218 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fac5805c2a0 msgr2=0x7fac5805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:58.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:58.169+0000 7fac74fa7700 1 --2- 172.21.15.58:0/3108632218 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fac5805c2a0 0x7fac5805e760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fac64009d60 tx=0x7fac64009b30).stop 2021-11-20T05:36:58.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:58.169+0000 7fac74fa7700 1 -- 172.21.15.58:0/3108632218 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fac700f5570 msgr2=0x7fac700604f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:58.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:58.169+0000 7fac74fa7700 1 --2- 172.21.15.58:0/3108632218 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fac700f5570 0x7fac700604f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fac60002700 tx=0x7fac60009d80).stop 2021-11-20T05:36:58.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:58.169+0000 7fac74fa7700 1 -- 172.21.15.58:0/3108632218 shutdown_connections 2021-11-20T05:36:58.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:58.169+0000 7fac74fa7700 1 --2- 172.21.15.58:0/3108632218 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fac5805c2a0 0x7fac5805e760 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:58.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:58.169+0000 7fac74fa7700 1 --2- 172.21.15.58:0/3108632218 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fac700f5570 0x7fac700604f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:58.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:58.169+0000 7fac74fa7700 1 --2- 172.21.15.58:0/3108632218 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac700f6810 0x7fac7010e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:58.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:58.169+0000 7fac74fa7700 1 -- 172.21.15.58:0/3108632218 >> 172.21.15.58:0/3108632218 conn(0x7fac700f0cf0 msgr2=0x7fac700fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:58.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:58.169+0000 7fac74fa7700 1 -- 172.21.15.58:0/3108632218 shutdown_connections 2021-11-20T05:36:58.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:58.169+0000 7fac74fa7700 1 -- 172.21.15.58:0/3108632218 wait complete. 2021-11-20T05:36:58.171 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 63 2021-11-20T05:36:58.180 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:49.273931+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24367, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24367': {'gid': 24367, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 59, 'state': 'up:rejoin', 'state_seq': 15, 'addr': '172.21.15.187:6825/126010257', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 126010257}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 126010257}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:58.180 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24367:up:rejoin 2021-11-20T05:36:58.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:58 smithi058 conmon[33537]: audit 2021-11-20T05:36:58.169369+0000 mon.smithi058 (mon.0) 701 : audit [DBG] 2021-11-20T05:36:58.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:58 smithi058 conmon[33537]: from='client.? 172.21.15.58:0/3108632218' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:58.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:58 smithi187 conmon[36358]: audit 2021-11-20T05:36:58.169369+0000 mon.smithi058 (mon.0) 701 : audit [DBG] from='client.? 172.21.15.58:0/3108632218' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:36:59.182 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:36:59.318 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.318+0000 7fccf1b6e700 1 -- 172.21.15.58:0/3142485912 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fccec0f6810 msgr2=0x7fccec0f6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:59.318 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.318+0000 7fccf1b6e700 1 --2- 172.21.15.58:0/3142485912 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fccec0f6810 0x7fccec0f6c70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fcce0004660 tx=0x7fcce0009b30).stop 2021-11-20T05:36:59.319 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.318+0000 7fccf1b6e700 1 -- 172.21.15.58:0/3142485912 shutdown_connections 2021-11-20T05:36:59.319 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.318+0000 7fccf1b6e700 1 --2- 172.21.15.58:0/3142485912 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fccec0f6810 0x7fccec0f6c70 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:59.319 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.318+0000 7fccf1b6e700 1 --2- 172.21.15.58:0/3142485912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccec0f5570 0x7fccec0f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:59.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.318+0000 7fccf1b6e700 1 -- 172.21.15.58:0/3142485912 >> 172.21.15.58:0/3142485912 conn(0x7fccec0f0cf0 msgr2=0x7fccec0f3110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:59.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.318+0000 7fccf1b6e700 1 -- 172.21.15.58:0/3142485912 shutdown_connections 2021-11-20T05:36:59.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.318+0000 7fccf1b6e700 1 -- 172.21.15.58:0/3142485912 wait complete. 2021-11-20T05:36:59.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.318+0000 7fccf1b6e700 1 Processor -- start 2021-11-20T05:36:59.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.319+0000 7fccf1b6e700 1 -- start start 2021-11-20T05:36:59.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.319+0000 7fccf1b6e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccec0f5570 0x7fccec060510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:59.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.319+0000 7fccf1b6e700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fccec0f6810 0x7fccec10bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:59.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.319+0000 7fccf1b6e700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec10c150 con 0x7fccec0f6810 2021-11-20T05:36:59.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.319+0000 7fccf1b6e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec10c290 con 0x7fccec0f5570 2021-11-20T05:36:59.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.319+0000 7fcceaffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fccec0f6810 0x7fccec10bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:59.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.319+0000 7fcceaffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fccec0f6810 0x7fccec10bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41402/0 (socket says 172.21.15.58:41402) 2021-11-20T05:36:59.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.319+0000 7fcceaffd700 1 -- 172.21.15.58:0/1591018189 learned_addr learned my addr 172.21.15.58:0/1591018189 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:36:59.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.319+0000 7fcceb7fe700 1 --2- 172.21.15.58:0/1591018189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccec0f5570 0x7fccec060510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:59.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.319+0000 7fcceaffd700 1 -- 172.21.15.58:0/1591018189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccec0f5570 msgr2=0x7fccec060510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:59.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.319+0000 7fcceaffd700 1 --2- 172.21.15.58:0/1591018189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccec0f5570 0x7fccec060510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:59.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.319+0000 7fcceaffd700 1 -- 172.21.15.58:0/1591018189 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce0005040 con 0x7fccec0f6810 2021-11-20T05:36:59.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.319+0000 7fcceb7fe700 1 --2- 172.21.15.58:0/1591018189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccec0f5570 0x7fccec060510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:36:59.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.319+0000 7fcceaffd700 1 --2- 172.21.15.58:0/1591018189 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fccec0f6810 0x7fccec10bc60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fcce0009830 tx=0x7fcce0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:59.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.319+0000 7fccd3fff700 1 -- 172.21.15.58:0/1591018189 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcce0018070 con 0x7fccec0f6810 2021-11-20T05:36:59.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.320+0000 7fccf1b6e700 1 -- 172.21.15.58:0/1591018189 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccec10c490 con 0x7fccec0f6810 2021-11-20T05:36:59.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.320+0000 7fccd3fff700 1 -- 172.21.15.58:0/1591018189 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcce0013410 con 0x7fccec0f6810 2021-11-20T05:36:59.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.320+0000 7fccd3fff700 1 -- 172.21.15.58:0/1591018189 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcce001c3c0 con 0x7fccec0f6810 2021-11-20T05:36:59.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.320+0000 7fccf1b6e700 1 -- 172.21.15.58:0/1591018189 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccec10c9f0 con 0x7fccec0f6810 2021-11-20T05:36:59.326 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.321+0000 7fccf1b6e700 1 -- 172.21.15.58:0/1591018189 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccec027a00 con 0x7fccec0f6810 2021-11-20T05:36:59.327 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.321+0000 7fccd3fff700 1 -- 172.21.15.58:0/1591018189 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcce001d420 con 0x7fccec0f6810 2021-11-20T05:36:59.327 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.321+0000 7fccd3fff700 1 --2- 172.21.15.58:0/1591018189 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fccd405c2a0 0x7fccd405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:36:59.327 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.321+0000 7fccd3fff700 1 -- 172.21.15.58:0/1591018189 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fcce0077fa0 con 0x7fccec0f6810 2021-11-20T05:36:59.327 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.321+0000 7fcceb7fe700 1 --2- 172.21.15.58:0/1591018189 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fccd405c2a0 0x7fccd405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:36:59.328 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.322+0000 7fcceb7fe700 1 --2- 172.21.15.58:0/1591018189 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fccd405c2a0 0x7fccd405e760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fccec0f6670 tx=0x7fccdc00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:36:59.328 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.325+0000 7fccd3fff700 1 -- 172.21.15.58:0/1591018189 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcce004a960 con 0x7fccec0f6810 2021-11-20T05:36:59.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:59 smithi058 conmon[33537]: cluster 2021-11-20T05:36:58.271740+0000 mgr.smithi058.zagdlk 2021-11-20T05:36:59.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:36:59 smithi058 conmon[33537]: (mgr.14164) 278 : cluster [DBG] pgmap v249: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:36:59.521 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.523+0000 7fccf1b6e700 1 -- 172.21.15.58:0/1591018189 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccec0fdd00 con 0x7fccec0f6810 2021-11-20T05:36:59.522 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.523+0000 7fccd3fff700 1 -- 172.21.15.58:0/1591018189 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2948 (secure 0 0 0) 0x7fcce004df80 con 0x7fccec0f6810 2021-11-20T05:36:59.523 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:36:59.523 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":63},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:49.273931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24367,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24367":{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":59,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:36:59.526 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.525+0000 7fccf1b6e700 1 -- 172.21.15.58:0/1591018189 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fccd405c2a0 msgr2=0x7fccd405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:59.526 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.525+0000 7fccf1b6e700 1 --2- 172.21.15.58:0/1591018189 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fccd405c2a0 0x7fccd405e760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fccec0f6670 tx=0x7fccdc00b040).stop 2021-11-20T05:36:59.526 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.525+0000 7fccf1b6e700 1 -- 172.21.15.58:0/1591018189 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fccec0f6810 msgr2=0x7fccec10bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:36:59.526 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.525+0000 7fccf1b6e700 1 --2- 172.21.15.58:0/1591018189 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fccec0f6810 0x7fccec10bc60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fcce0009830 tx=0x7fcce0009b30).stop 2021-11-20T05:36:59.527 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.525+0000 7fccf1b6e700 1 -- 172.21.15.58:0/1591018189 shutdown_connections 2021-11-20T05:36:59.527 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.525+0000 7fccf1b6e700 1 --2- 172.21.15.58:0/1591018189 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fccd405c2a0 0x7fccd405e760 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:59.528 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.525+0000 7fccf1b6e700 1 --2- 172.21.15.58:0/1591018189 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccec0f5570 0x7fccec060510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:59.528 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.525+0000 7fccf1b6e700 1 --2- 172.21.15.58:0/1591018189 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fccec0f6810 0x7fccec10bc60 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:36:59.528 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.525+0000 7fccf1b6e700 1 -- 172.21.15.58:0/1591018189 >> 172.21.15.58:0/1591018189 conn(0x7fccec0f0cf0 msgr2=0x7fccec0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:36:59.528 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.525+0000 7fccf1b6e700 1 -- 172.21.15.58:0/1591018189 shutdown_connections 2021-11-20T05:36:59.528 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:36:59.525+0000 7fccf1b6e700 1 -- 172.21.15.58:0/1591018189 wait complete. 2021-11-20T05:36:59.529 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 63 2021-11-20T05:36:59.535 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:49.273931+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24367, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24367': {'gid': 24367, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 59, 'state': 'up:rejoin', 'state_seq': 15, 'addr': '172.21.15.187:6825/126010257', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 126010257}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 126010257}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:36:59.536 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24367:up:rejoin 2021-11-20T05:36:59.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:36:59 smithi187 conmon[36358]: cluster 2021-11-20T05:36:58.271740+0000 mgr.smithi058.zagdlk (mgr.14164) 278 : cluster [DBG] pgmap v249: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:37:00.537 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:00.566 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:00 smithi058 conmon[33537]: audit 2021-11-20T05:36:59.525443+0000 mon.smithi058 (mon.0 2021-11-20T05:37:00.567 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:00 smithi058 conmon[33537]: ) 702 : audit [DBG] from='client.? 172.21.15.58:0/1591018189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:00.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.672+0000 7f0a04edb700 1 -- 172.21.15.58:0/3371803305 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0a000f66a0 msgr2=0x7f0a000f6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:00.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.672+0000 7f0a04edb700 1 --2- 172.21.15.58:0/3371803305 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0a000f66a0 0x7f0a000f6ac0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f09f80099e0 tx=0x7f09f8004d10).stop 2021-11-20T05:37:00.673 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.672+0000 7f0a04edb700 1 -- 172.21.15.58:0/3371803305 shutdown_connections 2021-11-20T05:37:00.673 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.672+0000 7f0a04edb700 1 --2- 172.21.15.58:0/3371803305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a000f7940 0x7f0a000f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:00.673 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.672+0000 7f0a04edb700 1 --2- 172.21.15.58:0/3371803305 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0a000f66a0 0x7f0a000f6ac0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:00.673 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.672+0000 7f0a04edb700 1 -- 172.21.15.58:0/3371803305 >> 172.21.15.58:0/3371803305 conn(0x7f0a000f1de0 msgr2=0x7f0a000f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:00.673 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.672+0000 7f0a04edb700 1 -- 172.21.15.58:0/3371803305 shutdown_connections 2021-11-20T05:37:00.674 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.672+0000 7f0a04edb700 1 -- 172.21.15.58:0/3371803305 wait complete. 2021-11-20T05:37:00.674 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.673+0000 7f0a04edb700 1 Processor -- start 2021-11-20T05:37:00.674 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.673+0000 7f0a04edb700 1 -- start start 2021-11-20T05:37:00.674 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.673+0000 7f0a04edb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a000f66a0 0x7f0a0006b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:00.675 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.673+0000 7f0a04edb700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0a000f7940 0x7f0a00069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:00.675 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.673+0000 7f0a04edb700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0006bd70 con 0x7f0a000f7940 2021-11-20T05:37:00.675 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.673+0000 7f0a04edb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0006a310 con 0x7f0a000f66a0 2021-11-20T05:37:00.676 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.673+0000 7f09fdd9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0a000f7940 0x7f0a00069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:00.676 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.673+0000 7f09fdd9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0a000f7940 0x7f0a00069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41412/0 (socket says 172.21.15.58:41412) 2021-11-20T05:37:00.676 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.673+0000 7f09fdd9b700 1 -- 172.21.15.58:0/1655716078 learned_addr learned my addr 172.21.15.58:0/1655716078 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:00.676 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.673+0000 7f09fe59c700 1 --2- 172.21.15.58:0/1655716078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a000f66a0 0x7f0a0006b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:00.677 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.673+0000 7f09fdd9b700 1 -- 172.21.15.58:0/1655716078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a000f66a0 msgr2=0x7f0a0006b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:00.677 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.673+0000 7f09fdd9b700 1 --2- 172.21.15.58:0/1655716078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a000f66a0 0x7f0a0006b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:00.677 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.673+0000 7f09fdd9b700 1 -- 172.21.15.58:0/1655716078 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f8009640 con 0x7f0a000f7940 2021-11-20T05:37:00.677 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.673+0000 7f09fdd9b700 1 --2- 172.21.15.58:0/1655716078 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0a000f7940 0x7f0a00069e20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f09f400c530 tx=0x7f09f4007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:00.677 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.673+0000 7f09eeffd700 1 -- 172.21.15.58:0/1655716078 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09f400f650 con 0x7f0a000f7940 2021-11-20T05:37:00.678 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.674+0000 7f09eeffd700 1 -- 172.21.15.58:0/1655716078 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09f400fc60 con 0x7f0a000f7940 2021-11-20T05:37:00.678 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.674+0000 7f0a04edb700 1 -- 172.21.15.58:0/1655716078 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0006a590 con 0x7f0a000f7940 2021-11-20T05:37:00.678 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.674+0000 7f09eeffd700 1 -- 172.21.15.58:0/1655716078 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09f4012c30 con 0x7f0a000f7940 2021-11-20T05:37:00.678 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.674+0000 7f0a04edb700 1 -- 172.21.15.58:0/1655716078 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0006aad0 con 0x7f0a000f7940 2021-11-20T05:37:00.679 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.675+0000 7f0a04edb700 1 -- 172.21.15.58:0/1655716078 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a00046330 con 0x7f0a000f7940 2021-11-20T05:37:00.680 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.675+0000 7f09eeffd700 1 -- 172.21.15.58:0/1655716078 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f09f4010070 con 0x7f0a000f7940 2021-11-20T05:37:00.680 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.675+0000 7f09eeffd700 1 --2- 172.21.15.58:0/1655716078 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f09e805c2a0 0x7f09e805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:00.680 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.676+0000 7f09eeffd700 1 -- 172.21.15.58:0/1655716078 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f09f4075ef0 con 0x7f0a000f7940 2021-11-20T05:37:00.680 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.676+0000 7f09fe59c700 1 --2- 172.21.15.58:0/1655716078 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f09e805c2a0 0x7f09e805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:00.681 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.676+0000 7f09fe59c700 1 --2- 172.21.15.58:0/1655716078 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f09e805c2a0 0x7f09e805e760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f09f800b620 tx=0x7f09f8004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:00.681 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.679+0000 7f09eeffd700 1 -- 172.21.15.58:0/1655716078 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09f40452b0 con 0x7f0a000f7940 2021-11-20T05:37:00.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:00 smithi187 conmon[36358]: audit 2021-11-20T05:36:59.525443+0000 mon.smithi058 ( 2021-11-20T05:37:00.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:00 smithi187 conmon[36358]: mon.0) 702 : audit [DBG] from='client.? 172.21.15.58:0/1591018189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:00.874 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.876+0000 7f0a04edb700 1 -- 172.21.15.58:0/1655716078 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a00027a00 con 0x7f0a000f7940 2021-11-20T05:37:00.874 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.876+0000 7f09eeffd700 1 -- 172.21.15.58:0/1655716078 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2948 (secure 0 0 0) 0x7f09f40487f0 con 0x7f0a000f7940 2021-11-20T05:37:00.875 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:00.875 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":63},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:49.273931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24367,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24367":{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":59,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:00.877 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.878+0000 7f0a04edb700 1 -- 172.21.15.58:0/1655716078 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f09e805c2a0 msgr2=0x7f09e805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:00.877 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.878+0000 7f0a04edb700 1 --2- 172.21.15.58:0/1655716078 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f09e805c2a0 0x7f09e805e760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f09f800b620 tx=0x7f09f8004d10).stop 2021-11-20T05:37:00.877 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.878+0000 7f0a04edb700 1 -- 172.21.15.58:0/1655716078 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0a000f7940 msgr2=0x7f0a00069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:00.877 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.878+0000 7f0a04edb700 1 --2- 172.21.15.58:0/1655716078 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0a000f7940 0x7f0a00069e20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f09f400c530 tx=0x7f09f4007e20).stop 2021-11-20T05:37:00.878 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.878+0000 7f0a04edb700 1 -- 172.21.15.58:0/1655716078 shutdown_connections 2021-11-20T05:37:00.878 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.878+0000 7f0a04edb700 1 --2- 172.21.15.58:0/1655716078 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f09e805c2a0 0x7f09e805e760 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:00.878 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.878+0000 7f0a04edb700 1 --2- 172.21.15.58:0/1655716078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0a000f66a0 0x7f0a0006b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:00.878 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.878+0000 7f0a04edb700 1 --2- 172.21.15.58:0/1655716078 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0a000f7940 0x7f0a00069e20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:00.879 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.878+0000 7f0a04edb700 1 -- 172.21.15.58:0/1655716078 >> 172.21.15.58:0/1655716078 conn(0x7f0a000f1de0 msgr2=0x7f0a000fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:00.879 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.878+0000 7f0a04edb700 1 -- 172.21.15.58:0/1655716078 shutdown_connections 2021-11-20T05:37:00.879 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:00.878+0000 7f0a04edb700 1 -- 172.21.15.58:0/1655716078 wait complete. 2021-11-20T05:37:00.879 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 63 2021-11-20T05:37:00.888 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:49.273931+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24367, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24367': {'gid': 24367, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 59, 'state': 'up:rejoin', 'state_seq': 15, 'addr': '172.21.15.187:6825/126010257', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 126010257}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 126010257}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:00.888 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24367:up:rejoin 2021-11-20T05:37:01.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:01 smithi187 conmon[36358]: cluster 2021-11-20T05:37:00.272223 2021-11-20T05:37:01.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:01 smithi187 conmon[36358]: +0000 mgr.smithi058.zagdlk (mgr.14164) 279 : cluster [DBG] pgmap v250: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2021-11-20T05:37:01.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:01 smithi187 conmon[36358]: audit 2021-11-20T05:37 2021-11-20T05:37:01.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:01 smithi187 conmon[36358]: :00.390935+0000 mon.smithi058 (mon.0) 703 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:37:01.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:01 smithi187 conmon[36358]: audit 2021-11-20T05:37:00.391360 2021-11-20T05:37:01.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:01 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 704 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:37:01.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:01 smithi187 conmon[36358]: audit 2021-11-20T 2021-11-20T05:37:01.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:01 smithi187 conmon[36358]: 05:37:00.878226+0000 mon.smithi058 (mon.0) 705 : audit [DBG] 2021-11-20T05:37:01.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:01 smithi187 conmon[36358]: from='client.? 172.21.15.58:0/1655716078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:01.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:01 smithi058 conmon[33537]: cluster 2021-11-20T05:37:00.272223+0000 mgr.smithi058.zagdlk (mgr.14164) 279 : cluster [DBG] pgmap v250: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2021-11-20T05:37:01.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:01 smithi058 conmon[33537]: audit 2021-11-20T05:37:00.390935+0000 mon.smithi058 (mon.0) 703 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:37:01.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:01 smithi058 conmon[33537]: audit 2021-11-20T05: 2021-11-20T05:37:01.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:01 smithi058 conmon[33537]: 37:00.391360+0000 mon.smithi058 (mon.0) 704 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:37:01.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:01 smithi058 conmon[33537]: audit 2021-11-20T05:37:00.878226+0000 2021-11-20T05:37:01.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:01 smithi058 conmon[33537]: mon.smithi058 (mon.0) 705 : audit [DBG] from='client.? 172.21.15.58:0/1655716078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:01.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:01 smithi058 conmon[33537]: 2021-11-20T05:37:01.889 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:02.024 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.024+0000 7f7548c30700 1 -- 172.21.15.58:0/4006634636 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f75440f5580 msgr2=0x7f75440f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:02.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.024+0000 7f7548c30700 1 --2- 172.21.15.58:0/4006634636 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f75440f5580 0x7f75440f59a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f753c004660 tx=0x7f753c009b30).stop 2021-11-20T05:37:02.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.024+0000 7f7548c30700 1 -- 172.21.15.58:0/4006634636 shutdown_connections 2021-11-20T05:37:02.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.024+0000 7f7548c30700 1 --2- 172.21.15.58:0/4006634636 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75440f6820 0x7f75440f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:02.026 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.024+0000 7f7548c30700 1 --2- 172.21.15.58:0/4006634636 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f75440f5580 0x7f75440f59a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:02.026 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.024+0000 7f7548c30700 1 -- 172.21.15.58:0/4006634636 >> 172.21.15.58:0/4006634636 conn(0x7f75440f0ce0 msgr2=0x7f75440f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:02.026 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.024+0000 7f7548c30700 1 -- 172.21.15.58:0/4006634636 shutdown_connections 2021-11-20T05:37:02.026 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.024+0000 7f7548c30700 1 -- 172.21.15.58:0/4006634636 wait complete. 2021-11-20T05:37:02.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.025+0000 7f7548c30700 1 Processor -- start 2021-11-20T05:37:02.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.025+0000 7f7548c30700 1 -- start start 2021-11-20T05:37:02.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.025+0000 7f7548c30700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75440f5580 0x7f7544106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:02.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.025+0000 7f7548c30700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f75440f6820 0x7f75441073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:02.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.025+0000 7f7548c30700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7544107950 con 0x7f75440f6820 2021-11-20T05:37:02.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.025+0000 7f7548c30700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7544107a90 con 0x7f75440f5580 2021-11-20T05:37:02.029 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.025+0000 7f7541d9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f75440f6820 0x7f75441073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:02.029 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.025+0000 7f7541d9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f75440f6820 0x7f75441073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41424/0 (socket says 172.21.15.58:41424) 2021-11-20T05:37:02.029 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.025+0000 7f7541d9b700 1 -- 172.21.15.58:0/3834855416 learned_addr learned my addr 172.21.15.58:0/3834855416 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:02.029 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.025+0000 7f754259c700 1 --2- 172.21.15.58:0/3834855416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75440f5580 0x7f7544106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:02.030 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.025+0000 7f7541d9b700 1 -- 172.21.15.58:0/3834855416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75440f5580 msgr2=0x7f7544106eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:02.030 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.025+0000 7f7541d9b700 1 --2- 172.21.15.58:0/3834855416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75440f5580 0x7f7544106eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:02.030 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.025+0000 7f7541d9b700 1 -- 172.21.15.58:0/3834855416 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f753c005040 con 0x7f75440f6820 2021-11-20T05:37:02.030 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.026+0000 7f7541d9b700 1 --2- 172.21.15.58:0/3834855416 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f75440f6820 0x7f75441073a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f752c0026d0 tx=0x7f752c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:02.031 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.026+0000 7f753affd700 1 -- 172.21.15.58:0/3834855416 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f752c0079c0 con 0x7f75440f6820 2021-11-20T05:37:02.031 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.026+0000 7f753affd700 1 -- 172.21.15.58:0/3834855416 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f752c00f040 con 0x7f75440f6820 2021-11-20T05:37:02.031 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.026+0000 7f7548c30700 1 -- 172.21.15.58:0/3834855416 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754410c530 con 0x7f75440f6820 2021-11-20T05:37:02.032 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.026+0000 7f753affd700 1 -- 172.21.15.58:0/3834855416 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f752c0148b0 con 0x7f75440f6820 2021-11-20T05:37:02.032 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.026+0000 7f7548c30700 1 -- 172.21.15.58:0/3834855416 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754410caf0 con 0x7f75440f6820 2021-11-20T05:37:02.032 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.027+0000 7f7548c30700 1 -- 172.21.15.58:0/3834855416 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7544027a00 con 0x7f75440f6820 2021-11-20T05:37:02.033 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.027+0000 7f753affd700 1 -- 172.21.15.58:0/3834855416 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f752c007b20 con 0x7f75440f6820 2021-11-20T05:37:02.034 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.027+0000 7f753affd700 1 --2- 172.21.15.58:0/3834855416 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f753005c2a0 0x7f753005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:02.034 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.028+0000 7f753affd700 1 -- 172.21.15.58:0/3834855416 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f752c0761b0 con 0x7f75440f6820 2021-11-20T05:37:02.034 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.028+0000 7f754259c700 1 --2- 172.21.15.58:0/3834855416 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f753005c2a0 0x7f753005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:02.035 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.028+0000 7f754259c700 1 --2- 172.21.15.58:0/3834855416 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f753005c2a0 0x7f753005e760 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f753c009f50 tx=0x7f753c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:02.035 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.031+0000 7f753affd700 1 -- 172.21.15.58:0/3834855416 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f752c048b70 con 0x7f75440f6820 2021-11-20T05:37:02.225 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.227+0000 7f7548c30700 1 -- 172.21.15.58:0/3834855416 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75440fee10 con 0x7f75440f6820 2021-11-20T05:37:02.226 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.228+0000 7f753affd700 1 -- 172.21.15.58:0/3834855416 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2948 (secure 0 0 0) 0x7f752c04c190 con 0x7f75440f6820 2021-11-20T05:37:02.227 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:02.227 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":63},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:49.273931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24367,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24367":{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":59,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:02.229 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.230+0000 7f7548c30700 1 -- 172.21.15.58:0/3834855416 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f753005c2a0 msgr2=0x7f753005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:02.229 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.230+0000 7f7548c30700 1 --2- 172.21.15.58:0/3834855416 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f753005c2a0 0x7f753005e760 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f753c009f50 tx=0x7f753c009b30).stop 2021-11-20T05:37:02.229 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.230+0000 7f7548c30700 1 -- 172.21.15.58:0/3834855416 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f75440f6820 msgr2=0x7f75441073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:02.229 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.230+0000 7f7548c30700 1 --2- 172.21.15.58:0/3834855416 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f75440f6820 0x7f75441073a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f752c0026d0 tx=0x7f752c009de0).stop 2021-11-20T05:37:02.230 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.230+0000 7f7548c30700 1 -- 172.21.15.58:0/3834855416 shutdown_connections 2021-11-20T05:37:02.230 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.230+0000 7f7548c30700 1 --2- 172.21.15.58:0/3834855416 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f753005c2a0 0x7f753005e760 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:02.230 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.230+0000 7f7548c30700 1 --2- 172.21.15.58:0/3834855416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75440f5580 0x7f7544106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:02.231 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.230+0000 7f7548c30700 1 --2- 172.21.15.58:0/3834855416 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f75440f6820 0x7f75441073a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:02.231 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.230+0000 7f7548c30700 1 -- 172.21.15.58:0/3834855416 >> 172.21.15.58:0/3834855416 conn(0x7f75440f0ce0 msgr2=0x7f75440fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:02.232 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.230+0000 7f7548c30700 1 -- 172.21.15.58:0/3834855416 shutdown_connections 2021-11-20T05:37:02.232 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:02.230+0000 7f7548c30700 1 -- 172.21.15.58:0/3834855416 wait complete. 2021-11-20T05:37:02.232 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 63 2021-11-20T05:37:02.240 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:49.273931+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24367, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24367': {'gid': 24367, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 59, 'state': 'up:rejoin', 'state_seq': 15, 'addr': '172.21.15.187:6825/126010257', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 126010257}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 126010257}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:02.240 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24367:up:rejoin 2021-11-20T05:37:02.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:02 smithi187 conmon[36358]: audit 2021-11-20T05:37:02.230051+0000 mon.smithi058 2021-11-20T05:37:02.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:02 smithi187 conmon[36358]: (mon.0) 706 : audit [DBG] from='client.? 172.21.15.58:0/3834855416' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:02.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:02 smithi058 conmon[33537]: audit 2021-11-20T05:37:02.230051+0000 mon.smithi058 2021-11-20T05:37:02.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:02 smithi058 conmon[33537]: (mon.0) 706 : audit [DBG] from='client.? 172.21.15.58:0/3834855416' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:03.241 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:03.377 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.377+0000 7f7d05b39700 1 -- 172.21.15.58:0/2360267943 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7d000f55a0 msgr2=0x7f7d000f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:03.377 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.377+0000 7f7d05b39700 1 --2- 172.21.15.58:0/2360267943 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7d000f55a0 0x7f7d000f59c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f7cf80099e0 tx=0x7f7cf8004d10).stop 2021-11-20T05:37:03.378 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.377+0000 7f7d05b39700 1 -- 172.21.15.58:0/2360267943 shutdown_connections 2021-11-20T05:37:03.378 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.377+0000 7f7d05b39700 1 --2- 172.21.15.58:0/2360267943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d000f6840 0x7f7d000f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:03.378 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.377+0000 7f7d05b39700 1 --2- 172.21.15.58:0/2360267943 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7d000f55a0 0x7f7d000f59c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:03.379 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.377+0000 7f7d05b39700 1 -- 172.21.15.58:0/2360267943 >> 172.21.15.58:0/2360267943 conn(0x7f7d000f0ce0 msgr2=0x7f7d000f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:03.379 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.377+0000 7f7d05b39700 1 -- 172.21.15.58:0/2360267943 shutdown_connections 2021-11-20T05:37:03.379 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.377+0000 7f7d05b39700 1 -- 172.21.15.58:0/2360267943 wait complete. 2021-11-20T05:37:03.379 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.377+0000 7f7d05b39700 1 Processor -- start 2021-11-20T05:37:03.380 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.378+0000 7f7d05b39700 1 -- start start 2021-11-20T05:37:03.380 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.378+0000 7f7d05b39700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7d000f55a0 0x7f7d00105eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:03.380 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.378+0000 7f7d05b39700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d000f6840 0x7f7d001063a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:03.380 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.378+0000 7f7d05b39700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d00106950 con 0x7f7d000f55a0 2021-11-20T05:37:03.381 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.378+0000 7f7d05b39700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d00106a90 con 0x7f7d000f6840 2021-11-20T05:37:03.381 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.378+0000 7f7cff7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7d000f55a0 0x7f7d00105eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:03.381 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.378+0000 7f7cff7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7d000f55a0 0x7f7d00105eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41432/0 (socket says 172.21.15.58:41432) 2021-11-20T05:37:03.382 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.378+0000 7f7cff7fe700 1 -- 172.21.15.58:0/2203130700 learned_addr learned my addr 172.21.15.58:0/2203130700 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:03.382 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.378+0000 7f7cfeffd700 1 --2- 172.21.15.58:0/2203130700 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d000f6840 0x7f7d001063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:03.382 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.378+0000 7f7cff7fe700 1 -- 172.21.15.58:0/2203130700 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d000f6840 msgr2=0x7f7d001063a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:03.383 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.378+0000 7f7cff7fe700 1 --2- 172.21.15.58:0/2203130700 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d000f6840 0x7f7d001063a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:03.383 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.378+0000 7f7cff7fe700 1 -- 172.21.15.58:0/2203130700 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf8009640 con 0x7f7d000f55a0 2021-11-20T05:37:03.383 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.378+0000 7f7cff7fe700 1 --2- 172.21.15.58:0/2203130700 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7d000f55a0 0x7f7d00105eb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f7cf80078b0 tx=0x7f7cf8004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:03.383 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.379+0000 7f7cf7fff700 1 -- 172.21.15.58:0/2203130700 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cf800e030 con 0x7f7d000f55a0 2021-11-20T05:37:03.384 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.379+0000 7f7d05b39700 1 -- 172.21.15.58:0/2203130700 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d0010b4d0 con 0x7f7d000f55a0 2021-11-20T05:37:03.384 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.379+0000 7f7cf7fff700 1 -- 172.21.15.58:0/2203130700 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7cf8007d50 con 0x7f7d000f55a0 2021-11-20T05:37:03.384 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.379+0000 7f7cf7fff700 1 -- 172.21.15.58:0/2203130700 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cf8018070 con 0x7f7d000f55a0 2021-11-20T05:37:03.384 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.379+0000 7f7d05b39700 1 -- 172.21.15.58:0/2203130700 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d0005f450 con 0x7f7d000f55a0 2021-11-20T05:37:03.385 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.380+0000 7f7d05b39700 1 -- 172.21.15.58:0/2203130700 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d000fdd30 con 0x7f7d000f55a0 2021-11-20T05:37:03.386 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.380+0000 7f7cf7fff700 1 -- 172.21.15.58:0/2203130700 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7cf80137e0 con 0x7f7d000f55a0 2021-11-20T05:37:03.386 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.380+0000 7f7cf7fff700 1 --2- 172.21.15.58:0/2203130700 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7cec05e4b0 0x7f7cec060970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:03.386 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.381+0000 7f7cf7fff700 1 -- 172.21.15.58:0/2203130700 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f7cf8077f40 con 0x7f7d000f55a0 2021-11-20T05:37:03.386 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.381+0000 7f7cfeffd700 1 --2- 172.21.15.58:0/2203130700 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7cec05e4b0 0x7f7cec060970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:03.387 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.381+0000 7f7cfeffd700 1 --2- 172.21.15.58:0/2203130700 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7cec05e4b0 0x7f7cec060970 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f7ce8004c40 tx=0x7f7ce800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:03.387 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.384+0000 7f7cf7fff700 1 -- 172.21.15.58:0/2203130700 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7cf80a42b0 con 0x7f7d000f55a0 2021-11-20T05:37:03.575 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:03 smithi187 conmon[36358]: cluster 2021-11-20T05:37:02.272874+0000 mgr.smithi058.zagdlk (mgr.14164) 280 : cluster [DBG] pgmap v251: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:37:03.582 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.584+0000 7f7d05b39700 1 -- 172.21.15.58:0/2203130700 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d00060310 con 0x7f7d000f55a0 2021-11-20T05:37:03.582 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.584+0000 7f7cf7fff700 1 -- 172.21.15.58:0/2203130700 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2948 (secure 0 0 0) 0x7f7cf804a9b0 con 0x7f7d000f55a0 2021-11-20T05:37:03.583 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:03.583 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":63},{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:36:49.273931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24367,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24367":{"gid":24367,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":59,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.187:6825/126010257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":126010257},{"type":"v1","addr":"172.21.15.187:6825","nonce":126010257}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:03.584 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.587+0000 7f7d05b39700 1 -- 172.21.15.58:0/2203130700 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7cec05e4b0 msgr2=0x7f7cec060970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:03.585 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.587+0000 7f7d05b39700 1 --2- 172.21.15.58:0/2203130700 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7cec05e4b0 0x7f7cec060970 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f7ce8004c40 tx=0x7f7ce800b040).stop 2021-11-20T05:37:03.585 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.587+0000 7f7d05b39700 1 -- 172.21.15.58:0/2203130700 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7d000f55a0 msgr2=0x7f7d00105eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:03.585 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.587+0000 7f7d05b39700 1 --2- 172.21.15.58:0/2203130700 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7d000f55a0 0x7f7d00105eb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f7cf80078b0 tx=0x7f7cf8004d10).stop 2021-11-20T05:37:03.585 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.587+0000 7f7d05b39700 1 -- 172.21.15.58:0/2203130700 shutdown_connections 2021-11-20T05:37:03.586 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.587+0000 7f7d05b39700 1 --2- 172.21.15.58:0/2203130700 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7cec05e4b0 0x7f7cec060970 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:03.586 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.587+0000 7f7d05b39700 1 --2- 172.21.15.58:0/2203130700 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7d000f55a0 0x7f7d00105eb0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:03.587 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.587+0000 7f7d05b39700 1 --2- 172.21.15.58:0/2203130700 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7d000f6840 0x7f7d001063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:03.587 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.587+0000 7f7d05b39700 1 -- 172.21.15.58:0/2203130700 >> 172.21.15.58:0/2203130700 conn(0x7f7d000f0ce0 msgr2=0x7f7d000fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:03.587 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.587+0000 7f7d05b39700 1 -- 172.21.15.58:0/2203130700 shutdown_connections 2021-11-20T05:37:03.588 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:03.587+0000 7f7d05b39700 1 -- 172.21.15.58:0/2203130700 wait complete. 2021-11-20T05:37:03.588 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 63 2021-11-20T05:37:03.595 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:03 smithi058 conmon[33537]: cluster 2021-11-20T05:37:02.272874+0000 mgr.smithi058.zagdlk (mgr.14164) 280 : cluster 2021-11-20T05:37:03.595 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:03 smithi058 conmon[33537]: [DBG] pgmap v251: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:37:03.597 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:36:49.273931+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24367, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24367': {'gid': 24367, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 59, 'state': 'up:rejoin', 'state_seq': 15, 'addr': '172.21.15.187:6825/126010257', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 126010257}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 126010257}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:03.597 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24367:up:rejoin 2021-11-20T05:37:04.599 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:04.628 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:04 smithi058 conmon[33537]: audit 2021-11-20T05:37:03.586477+0000 mon.smithi058 (mon.0 2021-11-20T05:37:04.629 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:04 smithi058 conmon[33537]: ) 707 : audit [DBG] from='client.? 172.21.15.58:0/2203130700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:04.629 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:04 smithi058 conmon[33537]: cluster 2021-11-20T05:37:04.190222+0000 mon.smithi058 (mon.0) 708 : cluster [INF] daemon mds.cephfs.smithi187.wxduhu restarted 2021-11-20T05:37:04.629 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:04 smithi058 conmon[33537]: cluster 2021-11-20T05:37:04.195153+0000 mon.smithi058 (mon.0) 709 : cluster [DBG] osdmap e52: 6 total, 6 up, 6 in 2021-11-20T05:37:04.629 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:04 smithi058 conmon[33537]: cluster 2021-11 2021-11-20T05:37:04.629 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:04 smithi058 conmon[33537]: -20T05:37:04.255273+0000 mon.smithi058 (mon.0) 2021-11-20T05:37:04.630 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:04 smithi058 conmon[33537]: 710 : cluster [INF] Standby daemon mds.cephfs.smithi058.rxapzd assigned to filesystem cephfs as rank 0 2021-11-20T05:37:04.630 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:04 smithi058 conmon[33537]: cluster 2021-11-20T05:37 2021-11-20T05:37:04.630 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:04 smithi058 conmon[33537]: :04.259434+0000 mon.smithi058 (mon.0) 711 : cluster [DBG] mds.? [v2:172.21.15.187:6824/2715421480,v1:172.21.15.187:6825/2715421480] up:boot 2021-11-20T05:37:04.630 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:04 smithi058 conmon[33537]: cluster 2021-11-20T05:37 2021-11-20T05:37:04.631 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:04 smithi058 conmon[33537]: :04.259529+0000 mon.smithi058 (mon.0) 712 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:37:04.631 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:04 smithi058 conmon[33537]: audit 2021-11-20T05:37:04. 2021-11-20T05:37:04.631 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:04 smithi058 conmon[33537]: 259865+0000 mon.smithi058 (mon.0) 713 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.wxduhu"}]: dispatch 2021-11-20T05:37:04.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:04 smithi187 conmon[36358]: audit 2021-11-20T05:37:03.586477+0000 mon.smithi058 (mon.0) 707 : audit [DBG] from='client.? 172.21.15.58:0/2203130700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:04.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:04 smithi187 conmon[36358]: cluster 2021-11-20T05:37:04.190222+0000 mon.smithi058 (mon.0) 708 : 2021-11-20T05:37:04.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:04 smithi187 conmon[36358]: cluster [INF] daemon mds.cephfs.smithi187.wxduhu restarted 2021-11-20T05:37:04.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:04 smithi187 conmon[36358]: cluster 2021-11-20T05:37:04.195153+0000 mon.smithi058 (mon.0) 709 : cluster [DBG] osdmap e52: 6 total, 6 up, 6 in 2021-11-20T05:37:04.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:04 smithi187 conmon[36358]: cluster 2021-11-20T05:37:04.255273+0000 mon.smithi058 (mon.0) 710 : cluster [INF] Standby daemon mds.cephfs.smithi058.rxapzd assigned to filesystem cephfs as rank 0 2021-11-20T05:37:04.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:04 smithi187 conmon[36358]: cluster 2021-11-20T05:37:04.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:04 smithi187 conmon[36358]: 2021-11-20T05:37:04.259434+0000 mon.smithi058 (mon.0) 711 : cluster [DBG] mds.? [v2:172.21.15.187:6824/2715421480,v1:172.21.15.187:6825/2715421480] up:boot 2021-11-20T05:37:04.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:04 smithi187 conmon[36358]: cluster 2021-11-20T05:37:04. 2021-11-20T05:37:04.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:04 smithi187 conmon[36358]: 259529+0000 mon.smithi058 (mon.0) 712 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:37:04.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:04 smithi187 conmon[36358]: 2021-11-20T05:37:04.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:04 smithi187 conmon[36358]: audit 2021-11-20 2021-11-20T05:37:04.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:04 smithi187 conmon[36358]: T05:37:04.259865+0000 mon.smithi058 (mon.0) 713 : audit 2021-11-20T05:37:04.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:04 smithi187 conmon[36358]: [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.wxduhu"}]: dispatch 2021-11-20T05:37:04.734 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.735+0000 7fec9b59e700 1 -- 172.21.15.58:0/1584317553 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec9c0f6770 msgr2=0x7fec9c0f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:04.735 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.735+0000 7fec9b59e700 1 --2- 172.21.15.58:0/1584317553 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec9c0f6770 0x7fec9c0f6bd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fec84004660 tx=0x7fec84009b30).stop 2021-11-20T05:37:04.735 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.735+0000 7fec9b59e700 1 -- 172.21.15.58:0/1584317553 shutdown_connections 2021-11-20T05:37:04.735 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.735+0000 7fec9b59e700 1 --2- 172.21.15.58:0/1584317553 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec9c0f6770 0x7fec9c0f6bd0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:04.735 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.735+0000 7fec9b59e700 1 --2- 172.21.15.58:0/1584317553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec9c0f54d0 0x7fec9c0f58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:04.736 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.735+0000 7fec9b59e700 1 -- 172.21.15.58:0/1584317553 >> 172.21.15.58:0/1584317553 conn(0x7fec9c0f0c30 msgr2=0x7fec9c0f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:04.736 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.735+0000 7fec9b59e700 1 -- 172.21.15.58:0/1584317553 shutdown_connections 2021-11-20T05:37:04.736 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.735+0000 7fec9b59e700 1 -- 172.21.15.58:0/1584317553 wait complete. 2021-11-20T05:37:04.736 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.736+0000 7fec9b59e700 1 Processor -- start 2021-11-20T05:37:04.736 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.736+0000 7fec9b59e700 1 -- start start 2021-11-20T05:37:04.737 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.736+0000 7fec9b59e700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec9c0f54d0 0x7fec9c109a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:04.737 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.736+0000 7fec9b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec9c0f6770 0x7fec9c10bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:04.737 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.736+0000 7fec9b59e700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c060540 con 0x7fec9c0f54d0 2021-11-20T05:37:04.737 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.736+0000 7fec9b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c060680 con 0x7fec9c0f6770 2021-11-20T05:37:04.738 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.736+0000 7fec9a59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec9c0f54d0 0x7fec9c109a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:04.738 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.736+0000 7fec9a59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec9c0f54d0 0x7fec9c109a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41452/0 (socket says 172.21.15.58:41452) 2021-11-20T05:37:04.738 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.736+0000 7fec9a59c700 1 -- 172.21.15.58:0/1154827392 learned_addr learned my addr 172.21.15.58:0/1154827392 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:04.738 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.736+0000 7fec99d9b700 1 --2- 172.21.15.58:0/1154827392 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec9c0f6770 0x7fec9c10bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:04.738 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.737+0000 7fec9a59c700 1 -- 172.21.15.58:0/1154827392 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec9c0f6770 msgr2=0x7fec9c10bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:04.739 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.737+0000 7fec9a59c700 1 --2- 172.21.15.58:0/1154827392 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec9c0f6770 0x7fec9c10bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:04.739 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.737+0000 7fec9a59c700 1 -- 172.21.15.58:0/1154827392 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec84005040 con 0x7fec9c0f54d0 2021-11-20T05:37:04.739 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.737+0000 7fec99d9b700 1 --2- 172.21.15.58:0/1154827392 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec9c0f6770 0x7fec9c10bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:37:04.739 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.737+0000 7fec9a59c700 1 --2- 172.21.15.58:0/1154827392 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec9c0f54d0 0x7fec9c109a20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fec9400c530 tx=0x7fec94007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:04.740 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.737+0000 7fec92ffd700 1 -- 172.21.15.58:0/1154827392 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec9400f650 con 0x7fec9c0f54d0 2021-11-20T05:37:04.740 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.737+0000 7fec92ffd700 1 -- 172.21.15.58:0/1154827392 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec9400fc60 con 0x7fec9c0f54d0 2021-11-20T05:37:04.740 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.737+0000 7fec9b59e700 1 -- 172.21.15.58:0/1154827392 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec9c10c4d0 con 0x7fec9c0f54d0 2021-11-20T05:37:04.740 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.737+0000 7fec92ffd700 1 -- 172.21.15.58:0/1154827392 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec94012d20 con 0x7fec9c0f54d0 2021-11-20T05:37:04.740 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.737+0000 7fec9b59e700 1 -- 172.21.15.58:0/1154827392 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec9c10ca90 con 0x7fec9c0f54d0 2021-11-20T05:37:04.742 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.738+0000 7fec9b59e700 1 -- 172.21.15.58:0/1154827392 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec9c045ad0 con 0x7fec9c0f54d0 2021-11-20T05:37:04.742 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.738+0000 7fec92ffd700 1 -- 172.21.15.58:0/1154827392 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fec94019030 con 0x7fec9c0f54d0 2021-11-20T05:37:04.742 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.739+0000 7fec92ffd700 1 --2- 172.21.15.58:0/1154827392 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fec8805c300 0x7fec8805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:04.742 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.739+0000 7fec92ffd700 1 -- 172.21.15.58:0/1154827392 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fec94075f90 con 0x7fec9c0f54d0 2021-11-20T05:37:04.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.739+0000 7fec99d9b700 1 --2- 172.21.15.58:0/1154827392 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fec8805c300 0x7fec8805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:04.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.739+0000 7fec99d9b700 1 --2- 172.21.15.58:0/1154827392 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fec8805c300 0x7fec8805e7c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fec84013580 tx=0x7fec84009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:04.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.743+0000 7fec92ffd700 1 -- 172.21.15.58:0/1154827392 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec94048880 con 0x7fec9c0f54d0 2021-11-20T05:37:04.936 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.939+0000 7fec9b59e700 1 -- 172.21.15.58:0/1154827392 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec9c0fdd70 con 0x7fec9c0f54d0 2021-11-20T05:37:04.937 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.939+0000 7fec92ffd700 1 -- 172.21.15.58:0/1154827392 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2951 (secure 0 0 0) 0x7fec94017070 con 0x7fec9c0f54d0 2021-11-20T05:37:04.937 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:04.937 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":64}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:04.255342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14622,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14622":{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":64,"state":"up:replay","state_seq":2,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:04.939 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.941+0000 7fec9b59e700 1 -- 172.21.15.58:0/1154827392 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fec8805c300 msgr2=0x7fec8805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:04.939 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.941+0000 7fec9b59e700 1 --2- 172.21.15.58:0/1154827392 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fec8805c300 0x7fec8805e7c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fec84013580 tx=0x7fec84009b30).stop 2021-11-20T05:37:04.940 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.941+0000 7fec9b59e700 1 -- 172.21.15.58:0/1154827392 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec9c0f54d0 msgr2=0x7fec9c109a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:04.940 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.941+0000 7fec9b59e700 1 --2- 172.21.15.58:0/1154827392 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec9c0f54d0 0x7fec9c109a20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fec9400c530 tx=0x7fec94007e10).stop 2021-11-20T05:37:04.940 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.941+0000 7fec9b59e700 1 -- 172.21.15.58:0/1154827392 shutdown_connections 2021-11-20T05:37:04.940 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.941+0000 7fec9b59e700 1 --2- 172.21.15.58:0/1154827392 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fec8805c300 0x7fec8805e7c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:04.942 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.941+0000 7fec9b59e700 1 --2- 172.21.15.58:0/1154827392 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fec9c0f54d0 0x7fec9c109a20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:04.942 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.941+0000 7fec9b59e700 1 --2- 172.21.15.58:0/1154827392 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec9c0f6770 0x7fec9c10bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:04.942 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.941+0000 7fec9b59e700 1 -- 172.21.15.58:0/1154827392 >> 172.21.15.58:0/1154827392 conn(0x7fec9c0f0c30 msgr2=0x7fec9c0fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:04.943 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.941+0000 7fec9b59e700 1 -- 172.21.15.58:0/1154827392 shutdown_connections 2021-11-20T05:37:04.943 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:04.941+0000 7fec9b59e700 1 -- 172.21.15.58:0/1154827392 wait complete. 2021-11-20T05:37:04.943 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 64 2021-11-20T05:37:04.950 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:04.255342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14622, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14622': {'gid': 14622, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 64, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.58:6827/3191333321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 3191333321}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 3191333321}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:04.951 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14622:up:replay 2021-11-20T05:37:05.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:05 smithi187 conmon[36358]: cluster 2021-11-20T05:37:04.273249+0000 mgr.smithi058.zagdlk (mgr.14164) 281 : cluster [DBG] pgmap v253: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:37:05.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:05 smithi187 conmon[36358]: audit 2021-11-20T05:37:04.940465+0000 mon.smithi058 (mon.0) 714 : audit [DBG] from='client.? 172.21.15.58:0/1154827392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:05.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:05 smithi187 conmon[36358]: cluster 2021-11-20T05:37:05.267497+0000 mon.smithi058 (mon.0) 715 : cluster [DBG] mds.? [v2:172.21.15.58:6826/3191333321,v1:172.21.15.58:6827/3191333321] up:resolve 2021-11-20T05:37:05.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:05 smithi187 conmon[36358]: cluster 2021-11-20T05:37:05.267541+0000 mon.smithi058 (mon 2021-11-20T05:37:05.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:05 smithi187 conmon[36358]: .0) 716 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:37:05.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:05 smithi058 conmon[33537]: cluster 2021-11-20T05:37:04.273249+0000 mgr.smithi058.zagdlk (mgr.14164) 281 : cluster [DBG] pgmap v253: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:37:05.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:05 smithi058 conmon[33537]: audit 2021-11-20T05:37:04.940465+0000 mon.smithi058 (mon.0) 714 : audit [DBG] from='client.? 172.21.15.58:0/1154827392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:05.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:05 smithi058 conmon[33537]: cluster 2021-11-20T05:37:05.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:05 smithi058 conmon[33537]: 2021-11-20T05:37:05.267497+0000 mon.smithi058 2021-11-20T05:37:05.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:05 smithi058 conmon[33537]: (mon.0) 715 : cluster [DBG] mds.? [v2:172.21.15.58:6826/3191333321,v1:172.21.15.58:6827/3191333321] up:resolve 2021-11-20T05:37:05.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:05 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:37:05.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:05 smithi058 conmon[33537]: -11-20 2021-11-20T05:37:05.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:05 smithi058 conmon[33537]: T05:37:05.267541+0000 mon.smithi058 2021-11-20T05:37:05.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:05 smithi058 conmon[33537]: (mon.0) 716 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:37:05.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:05 smithi058 conmon[33537]: 2021-11-20T05:37:05.951 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:06.086 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.087+0000 7f7277f0a700 1 -- 172.21.15.58:0/736408830 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72700f5580 msgr2=0x7f72700f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:06.086 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.087+0000 7f7277f0a700 1 --2- 172.21.15.58:0/736408830 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72700f5580 0x7f72700f59a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f72680099e0 tx=0x7f7268004d10).stop 2021-11-20T05:37:06.087 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.087+0000 7f7277f0a700 1 -- 172.21.15.58:0/736408830 shutdown_connections 2021-11-20T05:37:06.087 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.087+0000 7f7277f0a700 1 --2- 172.21.15.58:0/736408830 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72700f6820 0x7f72700f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:06.087 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.087+0000 7f7277f0a700 1 --2- 172.21.15.58:0/736408830 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72700f5580 0x7f72700f59a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:06.087 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.087+0000 7f7277f0a700 1 -- 172.21.15.58:0/736408830 >> 172.21.15.58:0/736408830 conn(0x7f72700f0ce0 msgr2=0x7f72700f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:06.087 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.087+0000 7f7277f0a700 1 -- 172.21.15.58:0/736408830 shutdown_connections 2021-11-20T05:37:06.088 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.087+0000 7f7277f0a700 1 -- 172.21.15.58:0/736408830 wait complete. 2021-11-20T05:37:06.088 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.087+0000 7f7277f0a700 1 Processor -- start 2021-11-20T05:37:06.088 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.087+0000 7f7277f0a700 1 -- start start 2021-11-20T05:37:06.088 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f7277f0a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72700f5580 0x7f72700f9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:06.089 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f7277f0a700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72700f6820 0x7f72700f7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:06.089 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f7277f0a700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72700f9ab0 con 0x7f72700f6820 2021-11-20T05:37:06.089 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f7277f0a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72700f8050 con 0x7f72700f5580 2021-11-20T05:37:06.089 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f72754a5700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72700f6820 0x7f72700f7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:06.090 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f72754a5700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72700f6820 0x7f72700f7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41466/0 (socket says 172.21.15.58:41466) 2021-11-20T05:37:06.090 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f72754a5700 1 -- 172.21.15.58:0/3706096192 learned_addr learned my addr 172.21.15.58:0/3706096192 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:06.090 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f7275ca6700 1 --2- 172.21.15.58:0/3706096192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72700f5580 0x7f72700f9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:06.090 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f72754a5700 1 -- 172.21.15.58:0/3706096192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72700f5580 msgr2=0x7f72700f9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:06.090 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f72754a5700 1 --2- 172.21.15.58:0/3706096192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72700f5580 0x7f72700f9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:06.091 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f72754a5700 1 -- 172.21.15.58:0/3706096192 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7268009640 con 0x7f72700f6820 2021-11-20T05:37:06.091 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f7275ca6700 1 --2- 172.21.15.58:0/3706096192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72700f5580 0x7f72700f9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:37:06.091 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f72754a5700 1 --2- 172.21.15.58:0/3706096192 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72700f6820 0x7f72700f7b60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f726400c530 tx=0x7f7264007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:06.091 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f726e7fc700 1 -- 172.21.15.58:0/3706096192 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f726400f650 con 0x7f72700f6820 2021-11-20T05:37:06.092 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f726e7fc700 1 -- 172.21.15.58:0/3706096192 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f726400fc60 con 0x7f72700f6820 2021-11-20T05:37:06.092 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f726e7fc700 1 -- 172.21.15.58:0/3706096192 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7264012c80 con 0x7f72700f6820 2021-11-20T05:37:06.092 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f7277f0a700 1 -- 172.21.15.58:0/3706096192 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72700f82d0 con 0x7f72700f6820 2021-11-20T05:37:06.092 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.088+0000 7f7277f0a700 1 -- 172.21.15.58:0/3706096192 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72700f8810 con 0x7f72700f6820 2021-11-20T05:37:06.092 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.089+0000 7f7277f0a700 1 -- 172.21.15.58:0/3706096192 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7270046330 con 0x7f72700f6820 2021-11-20T05:37:06.094 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.090+0000 7f726e7fc700 1 -- 172.21.15.58:0/3706096192 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7264010070 con 0x7f72700f6820 2021-11-20T05:37:06.094 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.090+0000 7f726e7fc700 1 --2- 172.21.15.58:0/3706096192 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f725805c2a0 0x7f725805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:06.094 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.090+0000 7f726e7fc700 1 -- 172.21.15.58:0/3706096192 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f7264075f60 con 0x7f72700f6820 2021-11-20T05:37:06.094 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.090+0000 7f7275ca6700 1 --2- 172.21.15.58:0/3706096192 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f725805c2a0 0x7f725805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:06.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.090+0000 7f7275ca6700 1 --2- 172.21.15.58:0/3706096192 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f725805c2a0 0x7f725805e760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f726800b3b0 tx=0x7f7268004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:06.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.094+0000 7f726e7fc700 1 -- 172.21.15.58:0/3706096192 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72640452b0 con 0x7f72700f6820 2021-11-20T05:37:06.307 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.310+0000 7f7277f0a700 1 -- 172.21.15.58:0/3706096192 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72700fdd10 con 0x7f72700f6820 2021-11-20T05:37:06.308 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.311+0000 7f726e7fc700 1 -- 172.21.15.58:0/3706096192 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2954 (secure 0 0 0) 0x7f726404c2f0 con 0x7f72700f6820 2021-11-20T05:37:06.309 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:06.309 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":64}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:06.265812+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14622,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14622":{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":64,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:06.311 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.313+0000 7f7277f0a700 1 -- 172.21.15.58:0/3706096192 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f725805c2a0 msgr2=0x7f725805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:06.311 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.313+0000 7f7277f0a700 1 --2- 172.21.15.58:0/3706096192 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f725805c2a0 0x7f725805e760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f726800b3b0 tx=0x7f7268004d10).stop 2021-11-20T05:37:06.311 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.313+0000 7f7277f0a700 1 -- 172.21.15.58:0/3706096192 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72700f6820 msgr2=0x7f72700f7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:06.312 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.313+0000 7f7277f0a700 1 --2- 172.21.15.58:0/3706096192 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72700f6820 0x7f72700f7b60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f726400c530 tx=0x7f7264007e20).stop 2021-11-20T05:37:06.312 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.313+0000 7f7277f0a700 1 -- 172.21.15.58:0/3706096192 shutdown_connections 2021-11-20T05:37:06.312 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.313+0000 7f7277f0a700 1 --2- 172.21.15.58:0/3706096192 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f725805c2a0 0x7f725805e760 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:06.312 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.313+0000 7f7277f0a700 1 --2- 172.21.15.58:0/3706096192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72700f5580 0x7f72700f9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:06.312 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.313+0000 7f7277f0a700 1 --2- 172.21.15.58:0/3706096192 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72700f6820 0x7f72700f7b60 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:06.313 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.313+0000 7f7277f0a700 1 -- 172.21.15.58:0/3706096192 >> 172.21.15.58:0/3706096192 conn(0x7f72700f0ce0 msgr2=0x7f7270060260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:06.313 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.313+0000 7f7277f0a700 1 -- 172.21.15.58:0/3706096192 shutdown_connections 2021-11-20T05:37:06.313 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:06.313+0000 7f7277f0a700 1 -- 172.21.15.58:0/3706096192 wait complete. 2021-11-20T05:37:06.314 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 66 2021-11-20T05:37:06.324 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:06.265812+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14622, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14622': {'gid': 14622, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 64, 'state': 'up:reconnect', 'state_seq': 7, 'addr': '172.21.15.58:6827/3191333321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 3191333321}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 3191333321}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:06.324 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14622:up:reconnect 2021-11-20T05:37:07.325 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:07.450 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.453+0000 7f3c555d8700 1 -- 172.21.15.58:0/2239909289 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3c500f55a0 msgr2=0x7f3c500f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:07.450 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.453+0000 7f3c555d8700 1 --2- 172.21.15.58:0/2239909289 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3c500f55a0 0x7f3c500f59c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f3c40004660 tx=0x7f3c40009b30).stop 2021-11-20T05:37:07.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.453+0000 7f3c555d8700 1 -- 172.21.15.58:0/2239909289 shutdown_connections 2021-11-20T05:37:07.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.453+0000 7f3c555d8700 1 --2- 172.21.15.58:0/2239909289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c500f6840 0x7f3c500f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:07.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.453+0000 7f3c555d8700 1 --2- 172.21.15.58:0/2239909289 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3c500f55a0 0x7f3c500f59c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:07.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.453+0000 7f3c555d8700 1 -- 172.21.15.58:0/2239909289 >> 172.21.15.58:0/2239909289 conn(0x7f3c500f0ce0 msgr2=0x7f3c500f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:07.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.453+0000 7f3c555d8700 1 -- 172.21.15.58:0/2239909289 shutdown_connections 2021-11-20T05:37:07.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.453+0000 7f3c555d8700 1 -- 172.21.15.58:0/2239909289 wait complete. 2021-11-20T05:37:07.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.453+0000 7f3c555d8700 1 Processor -- start 2021-11-20T05:37:07.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c555d8700 1 -- start start 2021-11-20T05:37:07.455 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c555d8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c500f55a0 0x7f3c5005ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:07.456 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c555d8700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3c500f6840 0x7f3c5010bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:07.456 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c555d8700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5010c1d0 con 0x7f3c500f6840 2021-11-20T05:37:07.456 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c555d8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5010c310 con 0x7f3c500f55a0 2021-11-20T05:37:07.457 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c4e7fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3c500f6840 0x7f3c5010bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:07.457 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c4e7fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3c500f6840 0x7f3c5010bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41480/0 (socket says 172.21.15.58:41480) 2021-11-20T05:37:07.457 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c4e7fc700 1 -- 172.21.15.58:0/3493341262 learned_addr learned my addr 172.21.15.58:0/3493341262 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:07.457 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c4effd700 1 --2- 172.21.15.58:0/3493341262 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c500f55a0 0x7f3c5005ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:07.458 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c4e7fc700 1 -- 172.21.15.58:0/3493341262 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c500f55a0 msgr2=0x7f3c5005ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:07.458 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c4e7fc700 1 --2- 172.21.15.58:0/3493341262 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c500f55a0 0x7f3c5005ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:07.458 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c4e7fc700 1 -- 172.21.15.58:0/3493341262 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c40005040 con 0x7f3c500f6840 2021-11-20T05:37:07.459 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c4effd700 1 --2- 172.21.15.58:0/3493341262 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c500f55a0 0x7f3c5005ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:37:07.459 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c4e7fc700 1 --2- 172.21.15.58:0/3493341262 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3c500f6840 0x7f3c5010bce0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f3c4400c530 tx=0x7f3c44007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:07.459 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c377fe700 1 -- 172.21.15.58:0/3493341262 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c4400f650 con 0x7f3c500f6840 2021-11-20T05:37:07.460 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c377fe700 1 -- 172.21.15.58:0/3493341262 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c4400fc60 con 0x7f3c500f6840 2021-11-20T05:37:07.460 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c555d8700 1 -- 172.21.15.58:0/3493341262 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5010c570 con 0x7f3c500f6840 2021-11-20T05:37:07.460 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c377fe700 1 -- 172.21.15.58:0/3493341262 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c44012c80 con 0x7f3c500f6840 2021-11-20T05:37:07.460 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.454+0000 7f3c555d8700 1 -- 172.21.15.58:0/3493341262 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5010cb00 con 0x7f3c500f6840 2021-11-20T05:37:07.461 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.455+0000 7f3c555d8700 1 -- 172.21.15.58:0/3493341262 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c50046330 con 0x7f3c500f6840 2021-11-20T05:37:07.461 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.456+0000 7f3c377fe700 1 -- 172.21.15.58:0/3493341262 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3c44010070 con 0x7f3c500f6840 2021-11-20T05:37:07.461 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.456+0000 7f3c377fe700 1 --2- 172.21.15.58:0/3493341262 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3c3805c2a0 0x7f3c3805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:07.461 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.456+0000 7f3c377fe700 1 -- 172.21.15.58:0/3493341262 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f3c44075f60 con 0x7f3c500f6840 2021-11-20T05:37:07.461 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.456+0000 7f3c4effd700 1 --2- 172.21.15.58:0/3493341262 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3c3805c2a0 0x7f3c3805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:07.462 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.456+0000 7f3c4effd700 1 --2- 172.21.15.58:0/3493341262 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3c3805c2a0 0x7f3c3805e760 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f3c40009d60 tx=0x7f3c40009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:07.462 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.460+0000 7f3c377fe700 1 -- 172.21.15.58:0/3493341262 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c440452b0 con 0x7f3c500f6840 2021-11-20T05:37:07.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:07 smithi058 conmon[33537]: cluster 2021-11-20T05:37:06 2021-11-20T05:37:07.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:07 smithi058 conmon[33537]: .269284+0000 mon.smithi058 (mon.0) 717 : cluster [DBG] mds.? [v2:172.21.15.58:6826/3191333321,v1:172.21.15.58:6827/3191333321] up:reconnect 2021-11-20T05:37:07.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:07 smithi058 conmon[33537]: cluster 2021-11-20T05:37:06. 2021-11-20T05:37:07.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:07 smithi058 conmon[33537]: 269357+0000 mon.smithi058 (mon.0) 718 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:37:07.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:07 smithi058 conmon[33537]: cluster 2021-11- 2021-11-20T05:37:07.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:07 smithi058 conmon[33537]: 20T05:37:06.277752+0000 mgr.smithi058.zagdlk 2021-11-20T05:37:07.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:07 smithi058 conmon[33537]: (mgr.14164) 282 : cluster [DBG] pgmap v254: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:37:07.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:07 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:37:07.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:07 smithi058 conmon[33537]: 11-20T05:37:06.312022 2021-11-20T05:37:07.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:07 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 719 : audit 2021-11-20T05:37:07.469 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:07 smithi058 conmon[33537]: [DBG] from='client.? 172.21.15.58:0/3706096192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:07.653 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.656+0000 7f3c555d8700 1 -- 172.21.15.58:0/3493341262 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c50027a00 con 0x7f3c500f6840 2021-11-20T05:37:07.654 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.657+0000 7f3c377fe700 1 -- 172.21.15.58:0/3493341262 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f3c4404c2f0 con 0x7f3c500f6840 2021-11-20T05:37:07.654 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:07.655 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":64}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:07.267431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14622,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14622":{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:07.657 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.658+0000 7f3c555d8700 1 -- 172.21.15.58:0/3493341262 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3c3805c2a0 msgr2=0x7f3c3805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:07.657 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.658+0000 7f3c555d8700 1 --2- 172.21.15.58:0/3493341262 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3c3805c2a0 0x7f3c3805e760 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f3c40009d60 tx=0x7f3c40009b30).stop 2021-11-20T05:37:07.657 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.658+0000 7f3c555d8700 1 -- 172.21.15.58:0/3493341262 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3c500f6840 msgr2=0x7f3c5010bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:07.658 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.658+0000 7f3c555d8700 1 --2- 172.21.15.58:0/3493341262 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3c500f6840 0x7f3c5010bce0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f3c4400c530 tx=0x7f3c44007e20).stop 2021-11-20T05:37:07.658 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.658+0000 7f3c555d8700 1 -- 172.21.15.58:0/3493341262 shutdown_connections 2021-11-20T05:37:07.659 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.658+0000 7f3c555d8700 1 --2- 172.21.15.58:0/3493341262 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3c3805c2a0 0x7f3c3805e760 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:07.659 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.658+0000 7f3c555d8700 1 --2- 172.21.15.58:0/3493341262 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c500f55a0 0x7f3c5005ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:07.660 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.658+0000 7f3c555d8700 1 --2- 172.21.15.58:0/3493341262 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3c500f6840 0x7f3c5010bce0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:07.660 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.658+0000 7f3c555d8700 1 -- 172.21.15.58:0/3493341262 >> 172.21.15.58:0/3493341262 conn(0x7f3c500f0ce0 msgr2=0x7f3c500fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:07.660 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.659+0000 7f3c555d8700 1 -- 172.21.15.58:0/3493341262 shutdown_connections 2021-11-20T05:37:07.661 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:07.659+0000 7f3c555d8700 1 -- 172.21.15.58:0/3493341262 wait complete. 2021-11-20T05:37:07.661 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 67 2021-11-20T05:37:07.667 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:07.267431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14622, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14622': {'gid': 14622, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 64, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/3191333321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 3191333321}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 3191333321}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:07.668 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14622:up:rejoin 2021-11-20T05:37:07.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:07 smithi187 conmon[36358]: cluster 2021-11-20T05:37:06.269284+0000 mon.smithi058 (mon.0) 717 : cluster [DBG] 2021-11-20T05:37:07.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:07 smithi187 conmon[36358]: mds.? [v2:172.21.15.58:6826/3191333321,v1:172.21.15.58:6827/3191333321] up:reconnect 2021-11-20T05:37:07.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:07 smithi187 conmon[36358]: cluster 2021-11-20T05:37:06.269357+0000 mon.smithi058 (mon.0) 718 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:37:07.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:07 smithi187 conmon[36358]: cluster 2021-11-20T05:37:06.277752+0000 mgr.smithi058.zagdlk (mgr.14164) 282 2021-11-20T05:37:07.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:07 smithi187 conmon[36358]: : cluster [DBG] pgmap v254: 65 pgs: 65 active+clean; 3.8 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:37:07.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:07 smithi187 conmon[36358]: audit 2021-11-20T05:37:06. 2021-11-20T05:37:07.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:07 smithi187 conmon[36358]: 312022+0000 mon.smithi058 (mon.0) 719 : audit [DBG] from='client.? 172.21.15.58:0/3706096192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:08.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:08 smithi058 conmon[33537]: cluster 2021-11-20T05:37:07.274983+0000 mon.smithi058 ( 2021-11-20T05:37:08.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:08 smithi058 conmon[33537]: mon.0) 720 : cluster [DBG] mds.? [v2:172.21.15.58:6826/3191333321,v1:172.21.15.58:6827/3191333321] up:rejoin 2021-11-20T05:37:08.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:08 smithi058 conmon[33537]: cluster 2021-11-20T05:37:07.275043+0000 mon.smithi058 (mon.0) 721 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:37:08.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:08 smithi058 conmon[33537]: audit 2021-11-20T05:37:07.657820+0000 mon.smithi058 (mon.0) 722 : audit [DBG] from='client.? 172.21.15.58:0/3493341262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:08.669 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:08.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:08 smithi187 conmon[36358]: cluster 2021-11-20T05:37:07.274983+0000 mon.smithi058 (mon.0) 720 : cluster [DBG] mds.? [v2:172.21.15.58:6826/3191333321,v1:172.21.15.58:6827/3191333321] up:rejoin 2021-11-20T05:37:08.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:08 smithi187 conmon[36358]: cluster 2021-11-20T05:37:07.275043+0000 mon.smithi058 (mon.0) 721 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:37:08.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:08 smithi187 conmon[36358]: audit 2021-11-20T05:37:07.657820+0000 mon.smithi058 (mon.0) 722 : audit 2021-11-20T05:37:08.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:08 smithi187 conmon[36358]: [DBG] from='client.? 172.21.15.58:0/3493341262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:08.798 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.800+0000 7f157d9e6700 1 -- 172.21.15.58:0/4078099041 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15780f55a0 msgr2=0x7f15780f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:08.799 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.800+0000 7f157d9e6700 1 --2- 172.21.15.58:0/4078099041 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15780f55a0 0x7f15780f59c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f15680099e0 tx=0x7f1568004d10).stop 2021-11-20T05:37:08.799 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.800+0000 7f157d9e6700 1 -- 172.21.15.58:0/4078099041 shutdown_connections 2021-11-20T05:37:08.799 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.800+0000 7f157d9e6700 1 --2- 172.21.15.58:0/4078099041 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15780f6840 0x7f15780f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:08.799 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.800+0000 7f157d9e6700 1 --2- 172.21.15.58:0/4078099041 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15780f55a0 0x7f15780f59c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:08.800 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.800+0000 7f157d9e6700 1 -- 172.21.15.58:0/4078099041 >> 172.21.15.58:0/4078099041 conn(0x7f15780f0ce0 msgr2=0x7f15780f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:08.800 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.800+0000 7f157d9e6700 1 -- 172.21.15.58:0/4078099041 shutdown_connections 2021-11-20T05:37:08.800 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.800+0000 7f157d9e6700 1 -- 172.21.15.58:0/4078099041 wait complete. 2021-11-20T05:37:08.800 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.800+0000 7f157d9e6700 1 Processor -- start 2021-11-20T05:37:08.801 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.800+0000 7f157d9e6700 1 -- start start 2021-11-20T05:37:08.801 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.801+0000 7f157d9e6700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15780f55a0 0x7f1578105eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:08.801 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.801+0000 7f157d9e6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15780f6840 0x7f15781063a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:08.801 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.801+0000 7f157d9e6700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1578106950 con 0x7f15780f55a0 2021-11-20T05:37:08.802 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.801+0000 7f157d9e6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1578106a90 con 0x7f15780f6840 2021-11-20T05:37:08.802 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.801+0000 7f1576ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15780f55a0 0x7f1578105eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:08.802 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.801+0000 7f1576ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15780f55a0 0x7f1578105eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41498/0 (socket says 172.21.15.58:41498) 2021-11-20T05:37:08.802 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.801+0000 7f1576ffd700 1 -- 172.21.15.58:0/950284766 learned_addr learned my addr 172.21.15.58:0/950284766 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:08.803 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.801+0000 7f15767fc700 1 --2- 172.21.15.58:0/950284766 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15780f6840 0x7f15781063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:08.803 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.801+0000 7f1576ffd700 1 -- 172.21.15.58:0/950284766 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15780f6840 msgr2=0x7f15781063a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:08.803 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.801+0000 7f1576ffd700 1 --2- 172.21.15.58:0/950284766 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15780f6840 0x7f15781063a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:08.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.801+0000 7f1576ffd700 1 -- 172.21.15.58:0/950284766 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1568009640 con 0x7f15780f55a0 2021-11-20T05:37:08.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.801+0000 7f1576ffd700 1 --2- 172.21.15.58:0/950284766 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15780f55a0 0x7f1578105eb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f15680078d0 tx=0x7f1568004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:08.805 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.801+0000 7f156f7fe700 1 -- 172.21.15.58:0/950284766 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f156800e070 con 0x7f15780f55a0 2021-11-20T05:37:08.805 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.801+0000 7f157d9e6700 1 -- 172.21.15.58:0/950284766 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157810b4d0 con 0x7f15780f55a0 2021-11-20T05:37:08.805 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.801+0000 7f156f7fe700 1 -- 172.21.15.58:0/950284766 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f156801d8d0 con 0x7f15780f55a0 2021-11-20T05:37:08.805 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.801+0000 7f156f7fe700 1 -- 172.21.15.58:0/950284766 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1568018070 con 0x7f15780f55a0 2021-11-20T05:37:08.806 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.801+0000 7f157d9e6700 1 -- 172.21.15.58:0/950284766 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f157810ba90 con 0x7f15780f55a0 2021-11-20T05:37:08.807 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.802+0000 7f157d9e6700 1 -- 172.21.15.58:0/950284766 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1578027a00 con 0x7f15780f55a0 2021-11-20T05:37:08.808 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.803+0000 7f156f7fe700 1 -- 172.21.15.58:0/950284766 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1568024060 con 0x7f15780f55a0 2021-11-20T05:37:08.808 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.803+0000 7f156f7fe700 1 --2- 172.21.15.58:0/950284766 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f156405c2a0 0x7f156405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:08.808 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.803+0000 7f156f7fe700 1 -- 172.21.15.58:0/950284766 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f15680779a0 con 0x7f15780f55a0 2021-11-20T05:37:08.809 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.803+0000 7f15767fc700 1 --2- 172.21.15.58:0/950284766 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f156405c2a0 0x7f156405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:08.809 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.804+0000 7f15767fc700 1 --2- 172.21.15.58:0/950284766 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f156405c2a0 0x7f156405e760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f1560007f80 tx=0x7f156000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:08.809 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:08.807+0000 7f156f7fe700 1 -- 172.21.15.58:0/950284766 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f156804a300 con 0x7f15780f55a0 2021-11-20T05:37:09.019 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:09.022+0000 7f157d9e6700 1 -- 172.21.15.58:0/950284766 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15780fdd30 con 0x7f15780f55a0 2021-11-20T05:37:09.020 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:09.022+0000 7f156f7fe700 1 -- 172.21.15.58:0/950284766 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2950 (secure 0 0 0) 0x7f156804d920 con 0x7f15780f55a0 2021-11-20T05:37:09.020 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:09.021 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:07.267431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14622,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14622":{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:09.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:09.024+0000 7f157d9e6700 1 -- 172.21.15.58:0/950284766 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f156405c2a0 msgr2=0x7f156405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:09.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:09.024+0000 7f157d9e6700 1 --2- 172.21.15.58:0/950284766 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f156405c2a0 0x7f156405e760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f1560007f80 tx=0x7f156000b040).stop 2021-11-20T05:37:09.024 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:09.024+0000 7f157d9e6700 1 -- 172.21.15.58:0/950284766 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15780f55a0 msgr2=0x7f1578105eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:09.024 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:09.024+0000 7f157d9e6700 1 --2- 172.21.15.58:0/950284766 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15780f55a0 0x7f1578105eb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f15680078d0 tx=0x7f1568004d10).stop 2021-11-20T05:37:09.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:09.024+0000 7f157d9e6700 1 -- 172.21.15.58:0/950284766 shutdown_connections 2021-11-20T05:37:09.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:09.024+0000 7f157d9e6700 1 --2- 172.21.15.58:0/950284766 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f156405c2a0 0x7f156405e760 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:09.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:09.024+0000 7f157d9e6700 1 --2- 172.21.15.58:0/950284766 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f15780f55a0 0x7f1578105eb0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:09.026 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:09.024+0000 7f157d9e6700 1 --2- 172.21.15.58:0/950284766 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15780f6840 0x7f15781063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:09.026 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:09.024+0000 7f157d9e6700 1 -- 172.21.15.58:0/950284766 >> 172.21.15.58:0/950284766 conn(0x7f15780f0ce0 msgr2=0x7f15780fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:09.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:09.024+0000 7f157d9e6700 1 -- 172.21.15.58:0/950284766 shutdown_connections 2021-11-20T05:37:09.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:09.024+0000 7f157d9e6700 1 -- 172.21.15.58:0/950284766 wait complete. 2021-11-20T05:37:09.027 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 68 2021-11-20T05:37:09.033 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:07.267431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14622, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14622': {'gid': 14622, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 64, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/3191333321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 3191333321}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 3191333321}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:09.033 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14622:up:rejoin 2021-11-20T05:37:09.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:09 smithi187 conmon[36358]: cluster 2021-11-20T05:37:08.278834+0000 mon.smithi058 (mon.0) 723 : cluster [DBG] 2021-11-20T05:37:09.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:09 smithi187 conmon[36358]: mds.? [v2:172.21.15.187:6824/2715421480,v1:172.21.15.187:6825/2715421480] up:standby 2021-11-20T05:37:09.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:09 smithi187 conmon[36358]: cluster 2021-11-20T05:37:08.278899+0000 mon.smithi058 (mon.0) 724 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:37:09.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:09 smithi187 conmon[36358]: cluster 2021-11-20T05:37 2021-11-20T05:37:09.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:09 smithi187 conmon[36358]: :08.279270+0000 mgr.smithi058.zagdlk (mgr.14164) 283 : cluster [DBG] pgmap v255: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 306 B/s rd, 0 op/s 2021-11-20T05:37:09.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:09 smithi187 conmon[36358]: audit 2021-11-20T05:37:09.023231+0000 2021-11-20T05:37:09.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:09 smithi187 conmon[36358]: mon.smithi058 (mon.0) 725 : audit [DBG] from='client.? 172.21.15.58:0/950284766' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:09.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:09 smithi058 conmon[33537]: cluster 2021-11-20T05:37:08.278834+0000 mon.smithi058 (mon.0) 723 : cluster [DBG] mds.? [v2:172.21.15.187:6824/2715421480,v1:172.21.15.187:6825/2715421480] up:standby 2021-11-20T05:37:09.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:09 smithi058 conmon[33537]: cluster 2021-11-20T05:37:08.278899+0000 mon.smithi058 (mon.0) 724 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi058.rxapzd=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby 2021-11-20T05:37:09.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:09 smithi058 conmon[33537]: cluster 2021-11-20T05:37:08.279270+0000 mgr.smithi058.zagdlk (mgr.14164) 283 : cluster [DBG] pgmap v255: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 306 B/s rd, 0 op/s 2021-11-20T05:37:09.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:09 smithi058 conmon[33537]: audit 2021-11-20T05:37:09.023231+0000 mon.smithi058 (mon.0) 725 : audit [DBG] from='client.? 172.21.15.58:0/950284766' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:10.034 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:10.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.169+0000 7fc15682f700 1 -- 172.21.15.58:0/3421655417 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1500f65d0 msgr2=0x7fc1500f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:10.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.169+0000 7fc15682f700 1 --2- 172.21.15.58:0/3421655417 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1500f65d0 0x7fc1500f69f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fc14c004660 tx=0x7fc14c009b30).stop 2021-11-20T05:37:10.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.170+0000 7fc15682f700 1 -- 172.21.15.58:0/3421655417 shutdown_connections 2021-11-20T05:37:10.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.170+0000 7fc15682f700 1 --2- 172.21.15.58:0/3421655417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1500f7870 0x7fc1500f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:10.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.170+0000 7fc15682f700 1 --2- 172.21.15.58:0/3421655417 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1500f65d0 0x7fc1500f69f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:10.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.170+0000 7fc15682f700 1 -- 172.21.15.58:0/3421655417 >> 172.21.15.58:0/3421655417 conn(0x7fc1500f1d10 msgr2=0x7fc1500f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:10.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.170+0000 7fc15682f700 1 -- 172.21.15.58:0/3421655417 shutdown_connections 2021-11-20T05:37:10.172 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.170+0000 7fc15682f700 1 -- 172.21.15.58:0/3421655417 wait complete. 2021-11-20T05:37:10.172 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.170+0000 7fc15682f700 1 Processor -- start 2021-11-20T05:37:10.172 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.170+0000 7fc15682f700 1 -- start start 2021-11-20T05:37:10.172 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.170+0000 7fc15682f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1500f65d0 0x7fc150106d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:10.173 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.170+0000 7fc15682f700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1500f7870 0x7fc150107250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:10.173 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.170+0000 7fc15682f700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc150107850 con 0x7fc1500f7870 2021-11-20T05:37:10.173 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.170+0000 7fc15682f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc150108060 con 0x7fc1500f65d0 2021-11-20T05:37:10.173 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.170+0000 7fc15502c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1500f7870 0x7fc150107250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:10.174 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.170+0000 7fc15502c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1500f7870 0x7fc150107250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41512/0 (socket says 172.21.15.58:41512) 2021-11-20T05:37:10.174 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.171+0000 7fc15502c700 1 -- 172.21.15.58:0/1773691359 learned_addr learned my addr 172.21.15.58:0/1773691359 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:10.174 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.171+0000 7fc15582d700 1 --2- 172.21.15.58:0/1773691359 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1500f65d0 0x7fc150106d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:10.174 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.171+0000 7fc15502c700 1 -- 172.21.15.58:0/1773691359 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1500f65d0 msgr2=0x7fc150106d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:10.174 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.171+0000 7fc15502c700 1 --2- 172.21.15.58:0/1773691359 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1500f65d0 0x7fc150106d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:10.175 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.171+0000 7fc15502c700 1 -- 172.21.15.58:0/1773691359 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14c005040 con 0x7fc1500f7870 2021-11-20T05:37:10.175 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.171+0000 7fc15502c700 1 --2- 172.21.15.58:0/1773691359 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1500f7870 0x7fc150107250 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fc140002700 tx=0x7fc140009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:10.175 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.171+0000 7fc1467fc700 1 -- 172.21.15.58:0/1773691359 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc140009320 con 0x7fc1500f7870 2021-11-20T05:37:10.175 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.171+0000 7fc1467fc700 1 -- 172.21.15.58:0/1773691359 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc14000f040 con 0x7fc1500f7870 2021-11-20T05:37:10.176 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.171+0000 7fc15682f700 1 -- 172.21.15.58:0/1773691359 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1501082e0 con 0x7fc1500f7870 2021-11-20T05:37:10.176 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.171+0000 7fc1467fc700 1 -- 172.21.15.58:0/1773691359 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc140014820 con 0x7fc1500f7870 2021-11-20T05:37:10.176 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.171+0000 7fc15682f700 1 -- 172.21.15.58:0/1773691359 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1501088a0 con 0x7fc1500f7870 2021-11-20T05:37:10.176 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.172+0000 7fc15682f700 1 -- 172.21.15.58:0/1773691359 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1500f59c0 con 0x7fc1500f7870 2021-11-20T05:37:10.177 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.172+0000 7fc1467fc700 1 -- 172.21.15.58:0/1773691359 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc1400077b0 con 0x7fc1500f7870 2021-11-20T05:37:10.177 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.173+0000 7fc1467fc700 1 --2- 172.21.15.58:0/1773691359 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc13c05c300 0x7fc13c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:10.177 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.173+0000 7fc1467fc700 1 -- 172.21.15.58:0/1773691359 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fc1400765a0 con 0x7fc1500f7870 2021-11-20T05:37:10.177 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.173+0000 7fc15582d700 1 --2- 172.21.15.58:0/1773691359 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc13c05c300 0x7fc13c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:10.178 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.173+0000 7fc15582d700 1 --2- 172.21.15.58:0/1773691359 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc13c05c300 0x7fc13c05e7c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fc14c009d60 tx=0x7fc14c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:10.178 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.177+0000 7fc1467fc700 1 -- 172.21.15.58:0/1773691359 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc140048f00 con 0x7fc1500f7870 2021-11-20T05:37:10.385 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.389+0000 7fc15682f700 1 -- 172.21.15.58:0/1773691359 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc150045ad0 con 0x7fc1500f7870 2021-11-20T05:37:10.386 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.389+0000 7fc1467fc700 1 -- 172.21.15.58:0/1773691359 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2950 (secure 0 0 0) 0x7fc14004c520 con 0x7fc1500f7870 2021-11-20T05:37:10.389 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:10.389 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:07.267431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14622,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14622":{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:10.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.391+0000 7fc15682f700 1 -- 172.21.15.58:0/1773691359 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc13c05c300 msgr2=0x7fc13c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:10.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.391+0000 7fc15682f700 1 --2- 172.21.15.58:0/1773691359 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc13c05c300 0x7fc13c05e7c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fc14c009d60 tx=0x7fc14c009b30).stop 2021-11-20T05:37:10.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.391+0000 7fc15682f700 1 -- 172.21.15.58:0/1773691359 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1500f7870 msgr2=0x7fc150107250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:10.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.391+0000 7fc15682f700 1 --2- 172.21.15.58:0/1773691359 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1500f7870 0x7fc150107250 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fc140002700 tx=0x7fc140009e00).stop 2021-11-20T05:37:10.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.391+0000 7fc15682f700 1 -- 172.21.15.58:0/1773691359 shutdown_connections 2021-11-20T05:37:10.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.391+0000 7fc15682f700 1 --2- 172.21.15.58:0/1773691359 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc13c05c300 0x7fc13c05e7c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:10.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.391+0000 7fc15682f700 1 --2- 172.21.15.58:0/1773691359 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1500f65d0 0x7fc150106d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:10.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.391+0000 7fc15682f700 1 --2- 172.21.15.58:0/1773691359 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc1500f7870 0x7fc150107250 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:10.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.391+0000 7fc15682f700 1 -- 172.21.15.58:0/1773691359 >> 172.21.15.58:0/1773691359 conn(0x7fc1500f1d10 msgr2=0x7fc1500fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:10.394 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.391+0000 7fc15682f700 1 -- 172.21.15.58:0/1773691359 shutdown_connections 2021-11-20T05:37:10.394 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:10.391+0000 7fc15682f700 1 -- 172.21.15.58:0/1773691359 wait complete. 2021-11-20T05:37:10.394 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 68 2021-11-20T05:37:10.401 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:07.267431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14622, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14622': {'gid': 14622, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 64, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/3191333321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 3191333321}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 3191333321}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:10.401 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14622:up:rejoin 2021-11-20T05:37:11.403 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:11.537 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.538+0000 7fc388ac2700 1 -- 172.21.15.58:0/3162997314 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3840f7920 msgr2=0x7fc3840f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:11.537 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.538+0000 7fc388ac2700 1 --2- 172.21.15.58:0/3162997314 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3840f7920 0x7fc3840f7d80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fc378004660 tx=0x7fc378009b30).stop 2021-11-20T05:37:11.538 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.538+0000 7fc388ac2700 1 -- 172.21.15.58:0/3162997314 shutdown_connections 2021-11-20T05:37:11.538 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.538+0000 7fc388ac2700 1 --2- 172.21.15.58:0/3162997314 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3840f7920 0x7fc3840f7d80 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:11.538 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.538+0000 7fc388ac2700 1 --2- 172.21.15.58:0/3162997314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3840f6680 0x7fc3840f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:11.538 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.538+0000 7fc388ac2700 1 -- 172.21.15.58:0/3162997314 >> 172.21.15.58:0/3162997314 conn(0x7fc3840f1de0 msgr2=0x7fc3840f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:11.539 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.538+0000 7fc388ac2700 1 -- 172.21.15.58:0/3162997314 shutdown_connections 2021-11-20T05:37:11.539 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.538+0000 7fc388ac2700 1 -- 172.21.15.58:0/3162997314 wait complete. 2021-11-20T05:37:11.539 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.539+0000 7fc388ac2700 1 Processor -- start 2021-11-20T05:37:11.539 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.539+0000 7fc388ac2700 1 -- start start 2021-11-20T05:37:11.540 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.539+0000 7fc388ac2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3840f6680 0x7fc384106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:11.540 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.539+0000 7fc388ac2700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3840f7920 0x7fc384107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:11.540 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.539+0000 7fc388ac2700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc384107930 con 0x7fc3840f7920 2021-11-20T05:37:11.540 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.539+0000 7fc388ac2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc384107a70 con 0x7fc3840f6680 2021-11-20T05:37:11.540 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.539+0000 7fc381d9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3840f7920 0x7fc384107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:11.541 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.539+0000 7fc381d9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3840f7920 0x7fc384107380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41526/0 (socket says 172.21.15.58:41526) 2021-11-20T05:37:11.541 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.539+0000 7fc381d9b700 1 -- 172.21.15.58:0/2176179937 learned_addr learned my addr 172.21.15.58:0/2176179937 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:11.541 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.539+0000 7fc38259c700 1 --2- 172.21.15.58:0/2176179937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3840f6680 0x7fc384106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:11.541 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.539+0000 7fc381d9b700 1 -- 172.21.15.58:0/2176179937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3840f6680 msgr2=0x7fc384106e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:11.541 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.539+0000 7fc381d9b700 1 --2- 172.21.15.58:0/2176179937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3840f6680 0x7fc384106e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:11.542 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.539+0000 7fc381d9b700 1 -- 172.21.15.58:0/2176179937 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc378005040 con 0x7fc3840f7920 2021-11-20T05:37:11.542 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.540+0000 7fc381d9b700 1 --2- 172.21.15.58:0/2176179937 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3840f7920 0x7fc384107380 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fc378009f50 tx=0x7fc378009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:11.542 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.540+0000 7fc372ffd700 1 -- 172.21.15.58:0/2176179937 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc378018070 con 0x7fc3840f7920 2021-11-20T05:37:11.542 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.540+0000 7fc388ac2700 1 -- 172.21.15.58:0/2176179937 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38410c4b0 con 0x7fc3840f7920 2021-11-20T05:37:11.543 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.540+0000 7fc372ffd700 1 -- 172.21.15.58:0/2176179937 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc378013410 con 0x7fc3840f7920 2021-11-20T05:37:11.543 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.540+0000 7fc372ffd700 1 -- 172.21.15.58:0/2176179937 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc37801c3c0 con 0x7fc3840f7920 2021-11-20T05:37:11.543 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.540+0000 7fc388ac2700 1 -- 172.21.15.58:0/2176179937 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38410ca70 con 0x7fc3840f7920 2021-11-20T05:37:11.543 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.541+0000 7fc388ac2700 1 -- 172.21.15.58:0/2176179937 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc384027a00 con 0x7fc3840f7920 2021-11-20T05:37:11.544 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.541+0000 7fc372ffd700 1 -- 172.21.15.58:0/2176179937 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc37801d420 con 0x7fc3840f7920 2021-11-20T05:37:11.545 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.542+0000 7fc372ffd700 1 --2- 172.21.15.58:0/2176179937 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc36c05c2a0 0x7fc36c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:11.545 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.542+0000 7fc372ffd700 1 -- 172.21.15.58:0/2176179937 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fc378077f60 con 0x7fc3840f7920 2021-11-20T05:37:11.545 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.542+0000 7fc38259c700 1 --2- 172.21.15.58:0/2176179937 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc36c05c2a0 0x7fc36c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:11.545 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.542+0000 7fc38259c700 1 --2- 172.21.15.58:0/2176179937 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc36c05c2a0 0x7fc36c05e760 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fc3840f7780 tx=0x7fc37400b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:11.545 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.545+0000 7fc372ffd700 1 -- 172.21.15.58:0/2176179937 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc37804a800 con 0x7fc3840f7920 2021-11-20T05:37:11.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:11 smithi187 conmon[36358]: cluster 2021-11-20T05:37:10.279681+0000 mgr.smithi058.zagdlk (mgr.14164) 284 : cluster [DBG] pgmap v256: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:11.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:11 smithi187 conmon[36358]: audit 2021-11-20T05:37:10.390104+0000 mon.smithi058 (mon.0) 726 : audit [DBG] from='client.? 172.21.15.58:0/1773691359' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:11.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:11 smithi058 conmon[33537]: cluster 2021-11-20T05:37:10.279681+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:37:11.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:11 smithi058 conmon[33537]: ) 284 : cluster [DBG] pgmap v256: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:11.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:11 smithi058 conmon[33537]: audit 2021-11-20T05:37:10.390104+0000 mon.smithi058 (mon.0) 726 : audit [DBG] from='client.? 172.21.15.58:0/1773691359' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:11.739 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.742+0000 7fc388ac2700 1 -- 172.21.15.58:0/2176179937 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc38410c640 con 0x7fc3840f7920 2021-11-20T05:37:11.740 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.743+0000 7fc372ffd700 1 -- 172.21.15.58:0/2176179937 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2950 (secure 0 0 0) 0x7fc38410c640 con 0x7fc3840f7920 2021-11-20T05:37:11.740 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:11.740 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:07.267431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14622,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14622":{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:11.742 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.744+0000 7fc388ac2700 1 -- 172.21.15.58:0/2176179937 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc36c05c2a0 msgr2=0x7fc36c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:11.742 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.744+0000 7fc388ac2700 1 --2- 172.21.15.58:0/2176179937 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc36c05c2a0 0x7fc36c05e760 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fc3840f7780 tx=0x7fc37400b040).stop 2021-11-20T05:37:11.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.744+0000 7fc388ac2700 1 -- 172.21.15.58:0/2176179937 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3840f7920 msgr2=0x7fc384107380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:11.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.744+0000 7fc388ac2700 1 --2- 172.21.15.58:0/2176179937 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3840f7920 0x7fc384107380 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fc378009f50 tx=0x7fc378009b30).stop 2021-11-20T05:37:11.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.744+0000 7fc388ac2700 1 -- 172.21.15.58:0/2176179937 shutdown_connections 2021-11-20T05:37:11.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.744+0000 7fc388ac2700 1 --2- 172.21.15.58:0/2176179937 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fc36c05c2a0 0x7fc36c05e760 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:11.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.744+0000 7fc388ac2700 1 --2- 172.21.15.58:0/2176179937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3840f6680 0x7fc384106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:11.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.744+0000 7fc388ac2700 1 --2- 172.21.15.58:0/2176179937 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fc3840f7920 0x7fc384107380 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:11.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.744+0000 7fc388ac2700 1 -- 172.21.15.58:0/2176179937 >> 172.21.15.58:0/2176179937 conn(0x7fc3840f1de0 msgr2=0x7fc3840fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:11.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.744+0000 7fc388ac2700 1 -- 172.21.15.58:0/2176179937 shutdown_connections 2021-11-20T05:37:11.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:11.744+0000 7fc388ac2700 1 -- 172.21.15.58:0/2176179937 wait complete. 2021-11-20T05:37:11.745 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 68 2021-11-20T05:37:11.753 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:07.267431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14622, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14622': {'gid': 14622, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 64, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/3191333321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 3191333321}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 3191333321}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:11.754 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14622:up:rejoin 2021-11-20T05:37:12.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:12 smithi187 conmon[36358]: audit 2021-11-20T05:37:11.743597+0000 mon.smithi058 (mon.0) 727 : audit [DBG] from='client.? 172.21.15.58:0/2176179937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:12.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:12 smithi058 conmon[33537]: audit 2021-11-20T05:37:11.743597+0000 mon.smithi058 (mon.0) 2021-11-20T05:37:12.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:12 smithi058 conmon[33537]: 727 : audit [DBG] from='client.? 172.21.15.58:0/2176179937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:12.755 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:12.891 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.891+0000 7f6ca572d700 1 -- 172.21.15.58:0/162681841 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6ca00f65d0 msgr2=0x7f6ca00f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:12.891 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.891+0000 7f6ca572d700 1 --2- 172.21.15.58:0/162681841 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6ca00f65d0 0x7f6ca00f69f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f6c94004660 tx=0x7f6c94009b30).stop 2021-11-20T05:37:12.892 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.891+0000 7f6ca572d700 1 -- 172.21.15.58:0/162681841 shutdown_connections 2021-11-20T05:37:12.892 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.891+0000 7f6ca572d700 1 --2- 172.21.15.58:0/162681841 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ca00f7870 0x7f6ca00f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:12.892 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.891+0000 7f6ca572d700 1 --2- 172.21.15.58:0/162681841 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6ca00f65d0 0x7f6ca00f69f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:12.892 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.891+0000 7f6ca572d700 1 -- 172.21.15.58:0/162681841 >> 172.21.15.58:0/162681841 conn(0x7f6ca00f1d10 msgr2=0x7f6ca00f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:12.893 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.891+0000 7f6ca572d700 1 -- 172.21.15.58:0/162681841 shutdown_connections 2021-11-20T05:37:12.893 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.891+0000 7f6ca572d700 1 -- 172.21.15.58:0/162681841 wait complete. 2021-11-20T05:37:12.893 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.891+0000 7f6ca572d700 1 Processor -- start 2021-11-20T05:37:12.893 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.891+0000 7f6ca572d700 1 -- start start 2021-11-20T05:37:12.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.892+0000 7f6ca572d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ca00f65d0 0x7f6ca006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:12.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.892+0000 7f6ca572d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6ca00f7870 0x7f6ca0069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:12.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.892+0000 7f6ca572d700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca006a380 con 0x7f6ca00f7870 2021-11-20T05:37:12.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.892+0000 7f6ca572d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca006a4c0 con 0x7f6ca00f65d0 2021-11-20T05:37:12.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.892+0000 7f6c9f7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6ca00f7870 0x7f6ca0069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:12.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.892+0000 7f6c9f7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6ca00f7870 0x7f6ca0069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41538/0 (socket says 172.21.15.58:41538) 2021-11-20T05:37:12.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.892+0000 7f6c9f7fe700 1 -- 172.21.15.58:0/2375244815 learned_addr learned my addr 172.21.15.58:0/2375244815 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:12.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.892+0000 7f6c9ffff700 1 --2- 172.21.15.58:0/2375244815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ca00f65d0 0x7f6ca006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:12.896 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.892+0000 7f6c9f7fe700 1 -- 172.21.15.58:0/2375244815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ca00f65d0 msgr2=0x7f6ca006b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:12.896 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.892+0000 7f6c9f7fe700 1 --2- 172.21.15.58:0/2375244815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ca00f65d0 0x7f6ca006b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:12.896 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.892+0000 7f6c9f7fe700 1 -- 172.21.15.58:0/2375244815 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c94005040 con 0x7f6ca00f7870 2021-11-20T05:37:12.896 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.892+0000 7f6c9f7fe700 1 --2- 172.21.15.58:0/2375244815 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6ca00f7870 0x7f6ca0069e20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f6c9000c530 tx=0x7f6c90007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:12.896 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.892+0000 7f6c9cff9700 1 -- 172.21.15.58:0/2375244815 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c9000f650 con 0x7f6ca00f7870 2021-11-20T05:37:12.897 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.892+0000 7f6c9cff9700 1 -- 172.21.15.58:0/2375244815 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c9000fc60 con 0x7f6ca00f7870 2021-11-20T05:37:12.897 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.892+0000 7f6ca572d700 1 -- 172.21.15.58:0/2375244815 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca006a7a0 con 0x7f6ca00f7870 2021-11-20T05:37:12.897 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.892+0000 7f6c9cff9700 1 -- 172.21.15.58:0/2375244815 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c90012c80 con 0x7f6ca00f7870 2021-11-20T05:37:12.897 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.892+0000 7f6ca572d700 1 -- 172.21.15.58:0/2375244815 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca010ed80 con 0x7f6ca00f7870 2021-11-20T05:37:12.897 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.893+0000 7f6ca572d700 1 -- 172.21.15.58:0/2375244815 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca0045ad0 con 0x7f6ca00f7870 2021-11-20T05:37:12.898 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.894+0000 7f6c9cff9700 1 -- 172.21.15.58:0/2375244815 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6c90010070 con 0x7f6ca00f7870 2021-11-20T05:37:12.898 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.894+0000 7f6c9cff9700 1 --2- 172.21.15.58:0/2375244815 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6c8805c300 0x7f6c8805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:12.898 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.894+0000 7f6c9cff9700 1 -- 172.21.15.58:0/2375244815 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f6c90075ea0 con 0x7f6ca00f7870 2021-11-20T05:37:12.898 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.894+0000 7f6c9ffff700 1 --2- 172.21.15.58:0/2375244815 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6c8805c300 0x7f6c8805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:12.899 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.895+0000 7f6c9ffff700 1 --2- 172.21.15.58:0/2375244815 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6c8805c300 0x7f6c8805e7c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6ca006b730 tx=0x7f6c94009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:12.900 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:12.898+0000 7f6c9cff9700 1 -- 172.21.15.58:0/2375244815 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c900452b0 con 0x7f6ca00f7870 2021-11-20T05:37:13.092 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:13.095+0000 7f6ca572d700 1 -- 172.21.15.58:0/2375244815 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ca00fecf0 con 0x7f6ca00f7870 2021-11-20T05:37:13.092 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:13.095+0000 7f6c9cff9700 1 -- 172.21.15.58:0/2375244815 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2950 (secure 0 0 0) 0x7f6c9004c2b0 con 0x7f6ca00f7870 2021-11-20T05:37:13.093 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:13.093 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:07.267431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14622,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14622":{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:13.094 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:13.097+0000 7f6ca572d700 1 -- 172.21.15.58:0/2375244815 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6c8805c300 msgr2=0x7f6c8805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:13.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:13.097+0000 7f6ca572d700 1 --2- 172.21.15.58:0/2375244815 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6c8805c300 0x7f6c8805e7c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6ca006b730 tx=0x7f6c94009b30).stop 2021-11-20T05:37:13.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:13.097+0000 7f6ca572d700 1 -- 172.21.15.58:0/2375244815 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6ca00f7870 msgr2=0x7f6ca0069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:13.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:13.097+0000 7f6ca572d700 1 --2- 172.21.15.58:0/2375244815 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6ca00f7870 0x7f6ca0069e20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f6c9000c530 tx=0x7f6c90007e20).stop 2021-11-20T05:37:13.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:13.097+0000 7f6ca572d700 1 -- 172.21.15.58:0/2375244815 shutdown_connections 2021-11-20T05:37:13.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:13.097+0000 7f6ca572d700 1 --2- 172.21.15.58:0/2375244815 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6c8805c300 0x7f6c8805e7c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:13.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:13.097+0000 7f6ca572d700 1 --2- 172.21.15.58:0/2375244815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ca00f65d0 0x7f6ca006b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:13.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:13.097+0000 7f6ca572d700 1 --2- 172.21.15.58:0/2375244815 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6ca00f7870 0x7f6ca0069e20 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:13.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:13.097+0000 7f6ca572d700 1 -- 172.21.15.58:0/2375244815 >> 172.21.15.58:0/2375244815 conn(0x7f6ca00f1d10 msgr2=0x7f6ca00fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:13.097 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:13.097+0000 7f6ca572d700 1 -- 172.21.15.58:0/2375244815 shutdown_connections 2021-11-20T05:37:13.097 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:13.097+0000 7f6ca572d700 1 -- 172.21.15.58:0/2375244815 wait complete. 2021-11-20T05:37:13.097 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 68 2021-11-20T05:37:13.106 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:07.267431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14622, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14622': {'gid': 14622, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 64, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/3191333321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 3191333321}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 3191333321}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:13.106 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14622:up:rejoin 2021-11-20T05:37:13.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:13 smithi058 conmon[33537]: cluster 2021-11-20T05:37:12.280336+0000 mgr.smithi058.zagdlk (mgr 2021-11-20T05:37:13.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:13 smithi058 conmon[33537]: .14164) 285 : cluster [DBG] pgmap v257: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:13.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:13 smithi058 conmon[33537]: audit 2021-11-20T05:37:13.096324+0000 2021-11-20T05:37:13.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:13 smithi058 conmon[33537]: mon.smithi058 (mon.0) 728 : audit [DBG] from='client.? 172.21.15.58:0/2375244815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:13.575 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:13 smithi187 conmon[36358]: cluster 2021-11-20T05:37:12.280336+0000 mgr.smithi058.zagdlk (mgr.14164) 285 : cluster [DBG] pgmap v257: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:13.576 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:13 smithi187 conmon[36358]: audit 2021-11-20T05:37:13.096324 2021-11-20T05:37:13.576 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:13 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 728 : audit [DBG] from='client.? 172.21.15.58:0/2375244815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:14.108 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:14.243 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.244+0000 7f11bd699700 1 -- 172.21.15.58:0/4213975604 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f11b80f6670 msgr2=0x7f11b80f6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:14.243 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.244+0000 7f11bd699700 1 --2- 172.21.15.58:0/4213975604 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f11b80f6670 0x7f11b80f6a90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f11a8004660 tx=0x7f11a8009b30).stop 2021-11-20T05:37:14.243 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.244+0000 7f11bd699700 1 -- 172.21.15.58:0/4213975604 shutdown_connections 2021-11-20T05:37:14.244 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.244+0000 7f11bd699700 1 --2- 172.21.15.58:0/4213975604 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11b80f7910 0x7f11b80f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:14.244 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.244+0000 7f11bd699700 1 --2- 172.21.15.58:0/4213975604 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f11b80f6670 0x7f11b80f6a90 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:14.244 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.244+0000 7f11bd699700 1 -- 172.21.15.58:0/4213975604 >> 172.21.15.58:0/4213975604 conn(0x7f11b80f1db0 msgr2=0x7f11b80f41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:14.244 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.244+0000 7f11bd699700 1 -- 172.21.15.58:0/4213975604 shutdown_connections 2021-11-20T05:37:14.245 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.244+0000 7f11bd699700 1 -- 172.21.15.58:0/4213975604 wait complete. 2021-11-20T05:37:14.245 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.245+0000 7f11bd699700 1 Processor -- start 2021-11-20T05:37:14.245 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.245+0000 7f11bd699700 1 -- start start 2021-11-20T05:37:14.245 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.245+0000 7f11bd699700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11b80f6670 0x7f11b8106e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:14.246 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.245+0000 7f11bd699700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f11b80f7910 0x7f11b8107350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:14.246 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.245+0000 7f11bd699700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b8107900 con 0x7f11b80f7910 2021-11-20T05:37:14.246 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.245+0000 7f11bd699700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b8107a40 con 0x7f11b80f6670 2021-11-20T05:37:14.246 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.245+0000 7f11b67fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f11b80f7910 0x7f11b8107350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:14.246 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.245+0000 7f11b67fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f11b80f7910 0x7f11b8107350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41550/0 (socket says 172.21.15.58:41550) 2021-11-20T05:37:14.247 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.245+0000 7f11b67fc700 1 -- 172.21.15.58:0/1392320850 learned_addr learned my addr 172.21.15.58:0/1392320850 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:14.247 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.245+0000 7f11b6ffd700 1 --2- 172.21.15.58:0/1392320850 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11b80f6670 0x7f11b8106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:14.247 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.246+0000 7f11b67fc700 1 -- 172.21.15.58:0/1392320850 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11b80f6670 msgr2=0x7f11b8106e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:14.247 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.246+0000 7f11b67fc700 1 --2- 172.21.15.58:0/1392320850 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11b80f6670 0x7f11b8106e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:14.248 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.246+0000 7f11b67fc700 1 -- 172.21.15.58:0/1392320850 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a8005040 con 0x7f11b80f7910 2021-11-20T05:37:14.248 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.246+0000 7f11b67fc700 1 --2- 172.21.15.58:0/1392320850 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f11b80f7910 0x7f11b8107350 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f11a0002700 tx=0x7f11a0009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:14.248 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.246+0000 7f11af7fe700 1 -- 172.21.15.58:0/1392320850 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11a0009320 con 0x7f11b80f7910 2021-11-20T05:37:14.248 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.246+0000 7f11af7fe700 1 -- 172.21.15.58:0/1392320850 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11a000f040 con 0x7f11b80f7910 2021-11-20T05:37:14.248 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.246+0000 7f11bd699700 1 -- 172.21.15.58:0/1392320850 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b810c4e0 con 0x7f11b80f7910 2021-11-20T05:37:14.249 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.246+0000 7f11af7fe700 1 -- 172.21.15.58:0/1392320850 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11a0014870 con 0x7f11b80f7910 2021-11-20T05:37:14.249 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.246+0000 7f11bd699700 1 -- 172.21.15.58:0/1392320850 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b810caa0 con 0x7f11b80f7910 2021-11-20T05:37:14.250 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.247+0000 7f11bd699700 1 -- 172.21.15.58:0/1392320850 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b8027a00 con 0x7f11b80f7910 2021-11-20T05:37:14.250 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.247+0000 7f11af7fe700 1 -- 172.21.15.58:0/1392320850 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f11a0009480 con 0x7f11b80f7910 2021-11-20T05:37:14.251 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.248+0000 7f11af7fe700 1 --2- 172.21.15.58:0/1392320850 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f11a405c2a0 0x7f11a405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:14.251 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.248+0000 7f11af7fe700 1 -- 172.21.15.58:0/1392320850 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f11a0076550 con 0x7f11b80f7910 2021-11-20T05:37:14.251 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.248+0000 7f11b6ffd700 1 --2- 172.21.15.58:0/1392320850 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f11a405c2a0 0x7f11a405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:14.251 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.248+0000 7f11b6ffd700 1 --2- 172.21.15.58:0/1392320850 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f11a405c2a0 0x7f11a405e760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f11a8009f50 tx=0x7f11a8009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:14.252 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.252+0000 7f11af7fe700 1 -- 172.21.15.58:0/1392320850 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11a0048e50 con 0x7f11b80f7910 2021-11-20T05:37:14.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.450+0000 7f11bd699700 1 -- 172.21.15.58:0/1392320850 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11b80fee00 con 0x7f11b80f7910 2021-11-20T05:37:14.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.450+0000 7f11af7fe700 1 -- 172.21.15.58:0/1392320850 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2950 (secure 0 0 0) 0x7f11a004c470 con 0x7f11b80f7910 2021-11-20T05:37:14.448 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:14.448 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:07.267431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14622,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14622":{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:14.450 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.452+0000 7f11bd699700 1 -- 172.21.15.58:0/1392320850 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f11a405c2a0 msgr2=0x7f11a405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:14.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.452+0000 7f11bd699700 1 --2- 172.21.15.58:0/1392320850 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f11a405c2a0 0x7f11a405e760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f11a8009f50 tx=0x7f11a8009b30).stop 2021-11-20T05:37:14.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.452+0000 7f11bd699700 1 -- 172.21.15.58:0/1392320850 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f11b80f7910 msgr2=0x7f11b8107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:14.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.452+0000 7f11bd699700 1 --2- 172.21.15.58:0/1392320850 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f11b80f7910 0x7f11b8107350 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f11a0002700 tx=0x7f11a0009e00).stop 2021-11-20T05:37:14.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.452+0000 7f11bd699700 1 -- 172.21.15.58:0/1392320850 shutdown_connections 2021-11-20T05:37:14.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.452+0000 7f11bd699700 1 --2- 172.21.15.58:0/1392320850 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f11a405c2a0 0x7f11a405e760 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:14.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.452+0000 7f11bd699700 1 --2- 172.21.15.58:0/1392320850 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11b80f6670 0x7f11b8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:14.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.452+0000 7f11bd699700 1 --2- 172.21.15.58:0/1392320850 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f11b80f7910 0x7f11b8107350 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:14.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.452+0000 7f11bd699700 1 -- 172.21.15.58:0/1392320850 >> 172.21.15.58:0/1392320850 conn(0x7f11b80f1db0 msgr2=0x7f11b80fb360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:14.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.452+0000 7f11bd699700 1 -- 172.21.15.58:0/1392320850 shutdown_connections 2021-11-20T05:37:14.454 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:14.452+0000 7f11bd699700 1 -- 172.21.15.58:0/1392320850 wait complete. 2021-11-20T05:37:14.454 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 68 2021-11-20T05:37:14.461 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:07.267431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14622, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14622': {'gid': 14622, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 64, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/3191333321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 3191333321}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 3191333321}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:14.462 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14622:up:rejoin 2021-11-20T05:37:15.463 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:15.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.598+0000 7fa802b8b700 1 -- 172.21.15.58:0/706976952 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa7fc0f65d0 msgr2=0x7fa7fc0f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:15.598 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.598+0000 7fa802b8b700 1 --2- 172.21.15.58:0/706976952 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa7fc0f65d0 0x7fa7fc0f69f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fa7f8004660 tx=0x7fa7f8009b30).stop 2021-11-20T05:37:15.598 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.599+0000 7fa802b8b700 1 -- 172.21.15.58:0/706976952 shutdown_connections 2021-11-20T05:37:15.598 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.599+0000 7fa802b8b700 1 --2- 172.21.15.58:0/706976952 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7fc0f7870 0x7fa7fc0f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:15.598 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.599+0000 7fa802b8b700 1 --2- 172.21.15.58:0/706976952 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa7fc0f65d0 0x7fa7fc0f69f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:15.599 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.599+0000 7fa802b8b700 1 -- 172.21.15.58:0/706976952 >> 172.21.15.58:0/706976952 conn(0x7fa7fc0f1d10 msgr2=0x7fa7fc0f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:15.599 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.599+0000 7fa802b8b700 1 -- 172.21.15.58:0/706976952 shutdown_connections 2021-11-20T05:37:15.600 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.599+0000 7fa802b8b700 1 -- 172.21.15.58:0/706976952 wait complete. 2021-11-20T05:37:15.600 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.599+0000 7fa802b8b700 1 Processor -- start 2021-11-20T05:37:15.600 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.599+0000 7fa802b8b700 1 -- start start 2021-11-20T05:37:15.600 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.599+0000 7fa802b8b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa7fc0f65d0 0x7fa7fc106da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:15.601 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.599+0000 7fa802b8b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7fc0f7870 0x7fa7fc107290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:15.601 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.599+0000 7fa802b8b700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc107890 con 0x7fa7fc0f65d0 2021-11-20T05:37:15.601 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.599+0000 7fa802b8b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc1080d0 con 0x7fa7fc0f7870 2021-11-20T05:37:15.601 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.599+0000 7fa801b89700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa7fc0f65d0 0x7fa7fc106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:15.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.599+0000 7fa801b89700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa7fc0f65d0 0x7fa7fc106da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41558/0 (socket says 172.21.15.58:41558) 2021-11-20T05:37:15.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.599+0000 7fa801b89700 1 -- 172.21.15.58:0/1717932474 learned_addr learned my addr 172.21.15.58:0/1717932474 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:15.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.600+0000 7fa801388700 1 --2- 172.21.15.58:0/1717932474 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7fc0f7870 0x7fa7fc107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:15.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.600+0000 7fa801b89700 1 -- 172.21.15.58:0/1717932474 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7fc0f7870 msgr2=0x7fa7fc107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:15.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.600+0000 7fa801b89700 1 --2- 172.21.15.58:0/1717932474 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7fc0f7870 0x7fa7fc107290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:15.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.600+0000 7fa801b89700 1 -- 172.21.15.58:0/1717932474 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f8005040 con 0x7fa7fc0f65d0 2021-11-20T05:37:15.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.600+0000 7fa801388700 1 --2- 172.21.15.58:0/1717932474 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7fc0f7870 0x7fa7fc107290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:37:15.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.600+0000 7fa801b89700 1 --2- 172.21.15.58:0/1717932474 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa7fc0f65d0 0x7fa7fc106da0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fa7f8004e30 tx=0x7fa7f8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:15.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.600+0000 7fa7f27fc700 1 -- 172.21.15.58:0/1717932474 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7f8018070 con 0x7fa7fc0f65d0 2021-11-20T05:37:15.604 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.600+0000 7fa7f27fc700 1 -- 172.21.15.58:0/1717932474 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa7f8013410 con 0x7fa7fc0f65d0 2021-11-20T05:37:15.604 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.600+0000 7fa802b8b700 1 -- 172.21.15.58:0/1717932474 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7fc108350 con 0x7fa7fc0f65d0 2021-11-20T05:37:15.604 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.600+0000 7fa7f27fc700 1 -- 172.21.15.58:0/1717932474 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7f801c3c0 con 0x7fa7fc0f65d0 2021-11-20T05:37:15.604 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.600+0000 7fa802b8b700 1 -- 172.21.15.58:0/1717932474 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7fc108830 con 0x7fa7fc0f65d0 2021-11-20T05:37:15.604 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.601+0000 7fa802b8b700 1 -- 172.21.15.58:0/1717932474 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7fc0f59c0 con 0x7fa7fc0f65d0 2021-11-20T05:37:15.606 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.601+0000 7fa7f27fc700 1 -- 172.21.15.58:0/1717932474 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa7f801d420 con 0x7fa7fc0f65d0 2021-11-20T05:37:15.606 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.602+0000 7fa7f27fc700 1 --2- 172.21.15.58:0/1717932474 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa7e805c300 0x7fa7e805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:15.606 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.602+0000 7fa7f27fc700 1 -- 172.21.15.58:0/1717932474 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fa7f8077e20 con 0x7fa7fc0f65d0 2021-11-20T05:37:15.607 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.602+0000 7fa801388700 1 --2- 172.21.15.58:0/1717932474 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa7e805c300 0x7fa7e805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:15.607 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.602+0000 7fa801388700 1 --2- 172.21.15.58:0/1717932474 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa7e805c300 0x7fa7e805e7c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa7fc106b30 tx=0x7fa7ec009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:15.607 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.606+0000 7fa7f27fc700 1 -- 172.21.15.58:0/1717932474 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7f804a780 con 0x7fa7fc0f65d0 2021-11-20T05:37:15.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:15 smithi187 conmon[36358]: cluster 2021-11-20T05:37:14.280749+0000 mgr.smithi058.zagdlk (mgr.14164) 286 : cluster [DBG] pgmap v258: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:15.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:15 smithi187 conmon[36358]: audit 2021-11-20T05:37:14.451495+0000 mon.smithi058 (mon.0) 729 : audit [DBG] from='client.? 172.21.15.58:0/1392320850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:15.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:15 smithi058 conmon[33537]: cluster 2021-11-20T05:37:14.280749+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:37:15.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:15 smithi058 conmon[33537]: mgr.14164) 286 : cluster [DBG] pgmap v258: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:15.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:15 smithi058 conmon[33537]: audit 2021-11-20T05:37:14.451495+0000 mon.smithi058 (mon.0) 729 : audit [DBG] from='client.? 172.21.15.58:0/1392320850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:15.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:15 smithi058 conmon[33537]: 2021-11-20T05:37:15.800 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.804+0000 7fa802b8b700 1 -- 172.21.15.58:0/1717932474 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7fc027b20 con 0x7fa7fc0f65d0 2021-11-20T05:37:15.801 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.804+0000 7fa7f27fc700 1 -- 172.21.15.58:0/1717932474 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2950 (secure 0 0 0) 0x7fa7f804dda0 con 0x7fa7fc0f65d0 2021-11-20T05:37:15.802 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:15.802 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:07.267431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14622,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14622":{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:15.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.806+0000 7fa802b8b700 1 -- 172.21.15.58:0/1717932474 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa7e805c300 msgr2=0x7fa7e805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:15.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.806+0000 7fa802b8b700 1 --2- 172.21.15.58:0/1717932474 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa7e805c300 0x7fa7e805e7c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa7fc106b30 tx=0x7fa7ec009be0).stop 2021-11-20T05:37:15.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.806+0000 7fa802b8b700 1 -- 172.21.15.58:0/1717932474 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa7fc0f65d0 msgr2=0x7fa7fc106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:15.805 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.806+0000 7fa802b8b700 1 --2- 172.21.15.58:0/1717932474 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa7fc0f65d0 0x7fa7fc106da0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fa7f8004e30 tx=0x7fa7f8009b30).stop 2021-11-20T05:37:15.805 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.806+0000 7fa802b8b700 1 -- 172.21.15.58:0/1717932474 shutdown_connections 2021-11-20T05:37:15.805 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.806+0000 7fa802b8b700 1 --2- 172.21.15.58:0/1717932474 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa7e805c300 0x7fa7e805e7c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:15.805 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.806+0000 7fa802b8b700 1 --2- 172.21.15.58:0/1717932474 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa7fc0f65d0 0x7fa7fc106da0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:15.806 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.806+0000 7fa802b8b700 1 --2- 172.21.15.58:0/1717932474 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7fc0f7870 0x7fa7fc107290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:15.806 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.806+0000 7fa802b8b700 1 -- 172.21.15.58:0/1717932474 >> 172.21.15.58:0/1717932474 conn(0x7fa7fc0f1d10 msgr2=0x7fa7fc0fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:15.806 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.806+0000 7fa802b8b700 1 -- 172.21.15.58:0/1717932474 shutdown_connections 2021-11-20T05:37:15.807 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:15.806+0000 7fa802b8b700 1 -- 172.21.15.58:0/1717932474 wait complete. 2021-11-20T05:37:15.807 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 68 2021-11-20T05:37:15.815 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:07.267431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14622, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14622': {'gid': 14622, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 64, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/3191333321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 3191333321}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 3191333321}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:15.815 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14622:up:rejoin 2021-11-20T05:37:16.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:16 smithi187 conmon[36358]: audit 2021-11-20T05:37:15.805520+0000 mon.smithi058 (mon.0) 730 : audit [DBG] from='client.? 172.21.15.58:0/1717932474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:16.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:16 smithi058 conmon[33537]: audit 2021-11-20T05:37:15.805520+0000 mon.smithi058 (mon.0) 2021-11-20T05:37:16.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:16 smithi058 conmon[33537]: 730 : audit [DBG] from='client.? 172.21.15.58:0/1717932474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:16.817 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:16.952 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.953+0000 7f9ebcab8700 1 -- 172.21.15.58:0/1723700477 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9eb80f5580 msgr2=0x7f9eb80f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:16.953 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.953+0000 7f9ebcab8700 1 --2- 172.21.15.58:0/1723700477 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9eb80f5580 0x7f9eb80f59a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f9ea8004660 tx=0x7f9ea8009b30).stop 2021-11-20T05:37:16.953 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.953+0000 7f9ebcab8700 1 -- 172.21.15.58:0/1723700477 shutdown_connections 2021-11-20T05:37:16.953 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.953+0000 7f9ebcab8700 1 --2- 172.21.15.58:0/1723700477 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9eb80f6820 0x7f9eb80f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:16.953 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.953+0000 7f9ebcab8700 1 --2- 172.21.15.58:0/1723700477 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9eb80f5580 0x7f9eb80f59a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:16.954 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.953+0000 7f9ebcab8700 1 -- 172.21.15.58:0/1723700477 >> 172.21.15.58:0/1723700477 conn(0x7f9eb80f0ce0 msgr2=0x7f9eb80f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:16.954 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.953+0000 7f9ebcab8700 1 -- 172.21.15.58:0/1723700477 shutdown_connections 2021-11-20T05:37:16.954 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.953+0000 7f9ebcab8700 1 -- 172.21.15.58:0/1723700477 wait complete. 2021-11-20T05:37:16.955 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.954+0000 7f9ebcab8700 1 Processor -- start 2021-11-20T05:37:16.955 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.954+0000 7f9ebcab8700 1 -- start start 2021-11-20T05:37:16.955 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.954+0000 7f9ebcab8700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9eb80f5580 0x7f9eb8105db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:16.956 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.954+0000 7f9ebcab8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9eb80f6820 0x7f9eb81062a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:16.956 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.954+0000 7f9ebcab8700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb8106850 con 0x7f9eb80f5580 2021-11-20T05:37:16.956 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.954+0000 7f9ebcab8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb8106990 con 0x7f9eb80f6820 2021-11-20T05:37:16.956 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.954+0000 7f9eb659c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9eb80f5580 0x7f9eb8105db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:16.957 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.954+0000 7f9eb659c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9eb80f5580 0x7f9eb8105db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41570/0 (socket says 172.21.15.58:41570) 2021-11-20T05:37:16.957 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.954+0000 7f9eb659c700 1 -- 172.21.15.58:0/1005985006 learned_addr learned my addr 172.21.15.58:0/1005985006 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:16.957 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.954+0000 7f9eb5d9b700 1 --2- 172.21.15.58:0/1005985006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9eb80f6820 0x7f9eb81062a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:16.957 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.954+0000 7f9eb659c700 1 -- 172.21.15.58:0/1005985006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9eb80f6820 msgr2=0x7f9eb81062a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:16.958 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.954+0000 7f9eb659c700 1 --2- 172.21.15.58:0/1005985006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9eb80f6820 0x7f9eb81062a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:16.958 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.955+0000 7f9eb659c700 1 -- 172.21.15.58:0/1005985006 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea8005040 con 0x7f9eb80f5580 2021-11-20T05:37:16.958 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.955+0000 7f9eb659c700 1 --2- 172.21.15.58:0/1005985006 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9eb80f5580 0x7f9eb8105db0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f9ea8004e30 tx=0x7f9ea8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:16.958 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.955+0000 7f9ea6ffd700 1 -- 172.21.15.58:0/1005985006 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ea8018070 con 0x7f9eb80f5580 2021-11-20T05:37:16.959 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.955+0000 7f9ea6ffd700 1 -- 172.21.15.58:0/1005985006 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ea8013410 con 0x7f9eb80f5580 2021-11-20T05:37:16.959 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.955+0000 7f9ebcab8700 1 -- 172.21.15.58:0/1005985006 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb810b3d0 con 0x7f9eb80f5580 2021-11-20T05:37:16.959 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.955+0000 7f9ea6ffd700 1 -- 172.21.15.58:0/1005985006 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ea801de20 con 0x7f9eb80f5580 2021-11-20T05:37:16.959 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.955+0000 7f9ebcab8700 1 -- 172.21.15.58:0/1005985006 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb810b990 con 0x7f9eb80f5580 2021-11-20T05:37:16.960 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.956+0000 7f9ebcab8700 1 -- 172.21.15.58:0/1005985006 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb8027a00 con 0x7f9eb80f5580 2021-11-20T05:37:16.961 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.956+0000 7f9ea6ffd700 1 -- 172.21.15.58:0/1005985006 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9ea801d420 con 0x7f9eb80f5580 2021-11-20T05:37:16.961 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.957+0000 7f9ea6ffd700 1 --2- 172.21.15.58:0/1005985006 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9ea005c2a0 0x7f9ea005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:16.961 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.957+0000 7f9ea6ffd700 1 -- 172.21.15.58:0/1005985006 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f9ea8077ea0 con 0x7f9eb80f5580 2021-11-20T05:37:16.962 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.957+0000 7f9eb5d9b700 1 --2- 172.21.15.58:0/1005985006 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9ea005c2a0 0x7f9ea005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:16.962 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.957+0000 7f9eb5d9b700 1 --2- 172.21.15.58:0/1005985006 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9ea005c2a0 0x7f9ea005e760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f9eac004c40 tx=0x7f9eac00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:16.962 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:16.961+0000 7f9ea6ffd700 1 -- 172.21.15.58:0/1005985006 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ea804a800 con 0x7f9eb80f5580 2021-11-20T05:37:17.152 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:17.156+0000 7f9ebcab8700 1 -- 172.21.15.58:0/1005985006 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9eb80fdd10 con 0x7f9eb80f5580 2021-11-20T05:37:17.153 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:17.156+0000 7f9ea6ffd700 1 -- 172.21.15.58:0/1005985006 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2950 (secure 0 0 0) 0x7f9ea804de20 con 0x7f9eb80f5580 2021-11-20T05:37:17.154 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:17.154 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:07.267431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14622,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14622":{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:17.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:17.158+0000 7f9ebcab8700 1 -- 172.21.15.58:0/1005985006 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9ea005c2a0 msgr2=0x7f9ea005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:17.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:17.158+0000 7f9ebcab8700 1 --2- 172.21.15.58:0/1005985006 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9ea005c2a0 0x7f9ea005e760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f9eac004c40 tx=0x7f9eac00b040).stop 2021-11-20T05:37:17.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:17.158+0000 7f9ebcab8700 1 -- 172.21.15.58:0/1005985006 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9eb80f5580 msgr2=0x7f9eb8105db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:17.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:17.158+0000 7f9ebcab8700 1 --2- 172.21.15.58:0/1005985006 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9eb80f5580 0x7f9eb8105db0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f9ea8004e30 tx=0x7f9ea8009b30).stop 2021-11-20T05:37:17.157 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:17.158+0000 7f9ebcab8700 1 -- 172.21.15.58:0/1005985006 shutdown_connections 2021-11-20T05:37:17.157 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:17.158+0000 7f9ebcab8700 1 --2- 172.21.15.58:0/1005985006 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9ea005c2a0 0x7f9ea005e760 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:17.159 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:17.158+0000 7f9ebcab8700 1 --2- 172.21.15.58:0/1005985006 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9eb80f5580 0x7f9eb8105db0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:17.159 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:17.158+0000 7f9ebcab8700 1 --2- 172.21.15.58:0/1005985006 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9eb80f6820 0x7f9eb81062a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:17.159 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:17.158+0000 7f9ebcab8700 1 -- 172.21.15.58:0/1005985006 >> 172.21.15.58:0/1005985006 conn(0x7f9eb80f0ce0 msgr2=0x7f9eb80fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:17.159 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:17.158+0000 7f9ebcab8700 1 -- 172.21.15.58:0/1005985006 shutdown_connections 2021-11-20T05:37:17.160 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:17.158+0000 7f9ebcab8700 1 -- 172.21.15.58:0/1005985006 wait complete. 2021-11-20T05:37:17.160 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 68 2021-11-20T05:37:17.167 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:07.267431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14622, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14622': {'gid': 14622, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 64, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/3191333321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 3191333321}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 3191333321}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:17.168 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14622:up:rejoin 2021-11-20T05:37:17.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:17 smithi058 conmon[33537]: cluster 2021-11-20T05:37:16.281198+0000 mgr.smithi058.zagdlk (mgr.14164) 287 : cluster [DBG] pgmap v259: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:37:17.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:17 smithi058 conmon[33537]: audit 2021-11-20T05:37:17.157355+0000 mon.smithi058 (mon.0) 731 : audit [DBG] from='client.? 172.21.15.58:0/1005985006' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:17.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:17 smithi187 conmon[36358]: cluster 2021-11-20T05:37:16.281198+0000 mgr.smithi058.zagdlk (mgr.14164) 287 : cluster [DBG] pgmap v259: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:37:17.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:17 smithi187 conmon[36358]: 2021-11-20T05:37:17.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:17 smithi187 conmon[36358]: audit 2021-11-20T05:37:17.157355+0000 mon.smithi058 (mon.0) 731 : audit [DBG] from='client.? 172.21.15.58:0/1005985006' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:18.169 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:18.307 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.307+0000 7f7bb44c6700 1 -- 172.21.15.58:0/2588639458 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7bac0f5570 msgr2=0x7f7bac0f5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:18.308 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.307+0000 7f7bb44c6700 1 --2- 172.21.15.58:0/2588639458 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7bac0f5570 0x7f7bac0f5990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f7ba8004660 tx=0x7f7ba8009b30).stop 2021-11-20T05:37:18.308 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.307+0000 7f7bb44c6700 1 -- 172.21.15.58:0/2588639458 shutdown_connections 2021-11-20T05:37:18.308 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.307+0000 7f7bb44c6700 1 --2- 172.21.15.58:0/2588639458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bac0f6810 0x7f7bac0f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:18.309 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.307+0000 7f7bb44c6700 1 --2- 172.21.15.58:0/2588639458 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7bac0f5570 0x7f7bac0f5990 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:18.309 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.307+0000 7f7bb44c6700 1 -- 172.21.15.58:0/2588639458 >> 172.21.15.58:0/2588639458 conn(0x7f7bac0f0cb0 msgr2=0x7f7bac0f30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:18.309 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.307+0000 7f7bb44c6700 1 -- 172.21.15.58:0/2588639458 shutdown_connections 2021-11-20T05:37:18.309 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.307+0000 7f7bb44c6700 1 -- 172.21.15.58:0/2588639458 wait complete. 2021-11-20T05:37:18.310 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.307+0000 7f7bb44c6700 1 Processor -- start 2021-11-20T05:37:18.310 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.307+0000 7f7bb44c6700 1 -- start start 2021-11-20T05:37:18.310 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7bb44c6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bac0f5570 0x7f7bac109020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:18.310 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7bb44c6700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7bac0f6810 0x7f7bac109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:18.310 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7bb44c6700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac109ac0 con 0x7f7bac0f6810 2021-11-20T05:37:18.311 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7bb44c6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac109c00 con 0x7f7bac0f5570 2021-11-20T05:37:18.311 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7bb1a61700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7bac0f6810 0x7f7bac109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:18.311 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7bb1a61700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7bac0f6810 0x7f7bac109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41582/0 (socket says 172.21.15.58:41582) 2021-11-20T05:37:18.311 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7bb1a61700 1 -- 172.21.15.58:0/2935873613 learned_addr learned my addr 172.21.15.58:0/2935873613 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:18.312 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7bb2262700 1 --2- 172.21.15.58:0/2935873613 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bac0f5570 0x7f7bac109020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:18.312 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7bb1a61700 1 -- 172.21.15.58:0/2935873613 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bac0f5570 msgr2=0x7f7bac109020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:18.312 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7bb1a61700 1 --2- 172.21.15.58:0/2935873613 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bac0f5570 0x7f7bac109020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:18.312 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7bb1a61700 1 -- 172.21.15.58:0/2935873613 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba8005040 con 0x7f7bac0f6810 2021-11-20T05:37:18.312 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7bb2262700 1 --2- 172.21.15.58:0/2935873613 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bac0f5570 0x7f7bac109020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:37:18.313 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7bb1a61700 1 --2- 172.21.15.58:0/2935873613 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7bac0f6810 0x7f7bac109510 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f7ba000e7f0 tx=0x7f7ba000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:18.313 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7b9effd700 1 -- 172.21.15.58:0/2935873613 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ba00092e0 con 0x7f7bac0f6810 2021-11-20T05:37:18.313 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7bb44c6700 1 -- 172.21.15.58:0/2935873613 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bac10e6a0 con 0x7f7bac0f6810 2021-11-20T05:37:18.313 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7b9effd700 1 -- 172.21.15.58:0/2935873613 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ba000f040 con 0x7f7bac0f6810 2021-11-20T05:37:18.314 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7b9effd700 1 -- 172.21.15.58:0/2935873613 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ba00146c0 con 0x7f7bac0f6810 2021-11-20T05:37:18.314 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.308+0000 7f7bb44c6700 1 -- 172.21.15.58:0/2935873613 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bac10ec60 con 0x7f7bac0f6810 2021-11-20T05:37:18.314 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.309+0000 7f7bb44c6700 1 -- 172.21.15.58:0/2935873613 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bac0fdd00 con 0x7f7bac0f6810 2021-11-20T05:37:18.314 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.310+0000 7f7b9effd700 1 -- 172.21.15.58:0/2935873613 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7ba001b030 con 0x7f7bac0f6810 2021-11-20T05:37:18.315 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.310+0000 7f7b9effd700 1 --2- 172.21.15.58:0/2935873613 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7b9805c2a0 0x7f7b9805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:18.315 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.310+0000 7f7b9effd700 1 -- 172.21.15.58:0/2935873613 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f7ba0075fd0 con 0x7f7bac0f6810 2021-11-20T05:37:18.315 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.310+0000 7f7bb2262700 1 --2- 172.21.15.58:0/2935873613 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7b9805c2a0 0x7f7b9805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:18.316 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.310+0000 7f7bb2262700 1 --2- 172.21.15.58:0/2935873613 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7b9805c2a0 0x7f7b9805e760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f7ba8004d80 tx=0x7f7ba800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:18.316 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.314+0000 7f7b9effd700 1 -- 172.21.15.58:0/2935873613 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ba0048980 con 0x7f7bac0f6810 2021-11-20T05:37:18.509 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.513+0000 7f7bb44c6700 1 -- 172.21.15.58:0/2935873613 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bac10f300 con 0x7f7bac0f6810 2021-11-20T05:37:18.510 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.513+0000 7f7b9effd700 1 -- 172.21.15.58:0/2935873613 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2950 (secure 0 0 0) 0x7f7ba0019020 con 0x7f7bac0f6810 2021-11-20T05:37:18.511 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:18.511 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:07.267431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14622,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14622":{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:18.512 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.515+0000 7f7bb44c6700 1 -- 172.21.15.58:0/2935873613 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7b9805c2a0 msgr2=0x7f7b9805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:18.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.515+0000 7f7bb44c6700 1 --2- 172.21.15.58:0/2935873613 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7b9805c2a0 0x7f7b9805e760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f7ba8004d80 tx=0x7f7ba800b040).stop 2021-11-20T05:37:18.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.515+0000 7f7bb44c6700 1 -- 172.21.15.58:0/2935873613 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7bac0f6810 msgr2=0x7f7bac109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:18.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.515+0000 7f7bb44c6700 1 --2- 172.21.15.58:0/2935873613 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7bac0f6810 0x7f7bac109510 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f7ba000e7f0 tx=0x7f7ba000b040).stop 2021-11-20T05:37:18.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.515+0000 7f7bb44c6700 1 -- 172.21.15.58:0/2935873613 shutdown_connections 2021-11-20T05:37:18.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.515+0000 7f7bb44c6700 1 --2- 172.21.15.58:0/2935873613 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7b9805c2a0 0x7f7b9805e760 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:18.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.515+0000 7f7bb44c6700 1 --2- 172.21.15.58:0/2935873613 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7bac0f5570 0x7f7bac109020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:18.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.515+0000 7f7bb44c6700 1 --2- 172.21.15.58:0/2935873613 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7bac0f6810 0x7f7bac109510 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:18.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.515+0000 7f7bb44c6700 1 -- 172.21.15.58:0/2935873613 >> 172.21.15.58:0/2935873613 conn(0x7f7bac0f0cb0 msgr2=0x7f7bac0602a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:18.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.515+0000 7f7bb44c6700 1 -- 172.21.15.58:0/2935873613 shutdown_connections 2021-11-20T05:37:18.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:18.515+0000 7f7bb44c6700 1 -- 172.21.15.58:0/2935873613 wait complete. 2021-11-20T05:37:18.515 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 68 2021-11-20T05:37:18.524 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:07.267431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14622, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14622': {'gid': 14622, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 64, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/3191333321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 3191333321}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 3191333321}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:18.524 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14622:up:rejoin 2021-11-20T05:37:19.525 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:19.659 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.662+0000 7f34f5437700 1 -- 172.21.15.58:0/2372740943 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f34f00f78a0 msgr2=0x7f34f00f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:19.659 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.662+0000 7f34f5437700 1 --2- 172.21.15.58:0/2372740943 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f34f00f78a0 0x7f34f00f7d00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f34e4004660 tx=0x7f34e4009b30).stop 2021-11-20T05:37:19.659 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.662+0000 7f34f5437700 1 -- 172.21.15.58:0/2372740943 shutdown_connections 2021-11-20T05:37:19.659 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.662+0000 7f34f5437700 1 --2- 172.21.15.58:0/2372740943 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f34f00f78a0 0x7f34f00f7d00 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:19.660 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.662+0000 7f34f5437700 1 --2- 172.21.15.58:0/2372740943 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34f00f6600 0x7f34f00f6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:19.660 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.662+0000 7f34f5437700 1 -- 172.21.15.58:0/2372740943 >> 172.21.15.58:0/2372740943 conn(0x7f34f00f1d40 msgr2=0x7f34f00f4180 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:19.660 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.662+0000 7f34f5437700 1 -- 172.21.15.58:0/2372740943 shutdown_connections 2021-11-20T05:37:19.660 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.662+0000 7f34f5437700 1 -- 172.21.15.58:0/2372740943 wait complete. 2021-11-20T05:37:19.663 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.663+0000 7f34f5437700 1 Processor -- start 2021-11-20T05:37:19.663 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.663+0000 7f34f5437700 1 -- start start 2021-11-20T05:37:19.664 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.663+0000 7f34f5437700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34f00f6600 0x7f34f0106d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:19.664 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.663+0000 7f34f5437700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f34f00f78a0 0x7f34f0107250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:19.664 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.663+0000 7f34f5437700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f0107850 con 0x7f34f00f78a0 2021-11-20T05:37:19.664 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.663+0000 7f34f5437700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f0108060 con 0x7f34f00f6600 2021-11-20T05:37:19.664 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.663+0000 7f34ee7fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f34f00f78a0 0x7f34f0107250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:19.665 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.663+0000 7f34ee7fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f34f00f78a0 0x7f34f0107250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41592/0 (socket says 172.21.15.58:41592) 2021-11-20T05:37:19.665 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.663+0000 7f34ee7fc700 1 -- 172.21.15.58:0/3122540023 learned_addr learned my addr 172.21.15.58:0/3122540023 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:19.665 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.663+0000 7f34eeffd700 1 --2- 172.21.15.58:0/3122540023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34f00f6600 0x7f34f0106d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:19.665 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.663+0000 7f34ee7fc700 1 -- 172.21.15.58:0/3122540023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34f00f6600 msgr2=0x7f34f0106d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:19.666 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.663+0000 7f34ee7fc700 1 --2- 172.21.15.58:0/3122540023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34f00f6600 0x7f34f0106d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:19.666 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.663+0000 7f34ee7fc700 1 -- 172.21.15.58:0/3122540023 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e4005040 con 0x7f34f00f78a0 2021-11-20T05:37:19.666 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.663+0000 7f34eeffd700 1 --2- 172.21.15.58:0/3122540023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34f00f6600 0x7f34f0106d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:37:19.666 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.663+0000 7f34ee7fc700 1 --2- 172.21.15.58:0/3122540023 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f34f00f78a0 0x7f34f0107250 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f34e4000c00 tx=0x7f34e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:19.667 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.663+0000 7f34d77fe700 1 -- 172.21.15.58:0/3122540023 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34e4019070 con 0x7f34f00f78a0 2021-11-20T05:37:19.667 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.664+0000 7f34f5437700 1 -- 172.21.15.58:0/3122540023 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f0108280 con 0x7f34f00f78a0 2021-11-20T05:37:19.668 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.664+0000 7f34d77fe700 1 -- 172.21.15.58:0/3122540023 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34e4007e70 con 0x7f34f00f78a0 2021-11-20T05:37:19.668 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.664+0000 7f34d77fe700 1 -- 172.21.15.58:0/3122540023 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34e4014d00 con 0x7f34f00f78a0 2021-11-20T05:37:19.668 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.664+0000 7f34f5437700 1 -- 172.21.15.58:0/3122540023 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f01087e0 con 0x7f34f00f78a0 2021-11-20T05:37:19.668 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.665+0000 7f34f5437700 1 -- 172.21.15.58:0/3122540023 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f00f59f0 con 0x7f34f00f78a0 2021-11-20T05:37:19.669 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.665+0000 7f34d77fe700 1 -- 172.21.15.58:0/3122540023 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f34e4025070 con 0x7f34f00f78a0 2021-11-20T05:37:19.669 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.665+0000 7f34d77fe700 1 --2- 172.21.15.58:0/3122540023 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f34d805c300 0x7f34d805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:19.669 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.665+0000 7f34d77fe700 1 -- 172.21.15.58:0/3122540023 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f34e4077290 con 0x7f34f00f78a0 2021-11-20T05:37:19.669 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.666+0000 7f34eeffd700 1 --2- 172.21.15.58:0/3122540023 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f34d805c300 0x7f34d805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:19.670 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.666+0000 7f34eeffd700 1 --2- 172.21.15.58:0/3122540023 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f34d805c300 0x7f34d805e7c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f34f0106ad0 tx=0x7f34e000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:19.670 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.670+0000 7f34d77fe700 1 -- 172.21.15.58:0/3122540023 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34e4049b90 con 0x7f34f00f78a0 2021-11-20T05:37:19.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:19 smithi187 conmon[36358]: cluster 2021-11-20T05:37:18.281738+0000 mgr.smithi058.zagdlk (mgr.14164) 288 : cluster [DBG] pgmap v260: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:37:19.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:19 smithi187 conmon[36358]: audit 2021-11-20T05:37:18.514489+0000 mon.smithi058 (mon.0) 732 : audit [DBG] from='client.? 172.21.15.58:0/2935873613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:19.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:19 smithi058 conmon[33537]: cluster 2021-11-20T05:37:18.281738+0000 mgr.smithi058.zagdlk (mgr.14164) 288 2021-11-20T05:37:19.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:19 smithi058 conmon[33537]: : cluster [DBG] pgmap v260: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:37:19.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:19 smithi058 conmon[33537]: audit 2021-11-20T05:37:18.514489+0000 mon.smithi058 (mon.0) 732 : audit [DBG] from='client.? 172.21.15.58:0/2935873613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:19.864 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.867+0000 7f34f5437700 1 -- 172.21.15.58:0/3122540023 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34f0045ad0 con 0x7f34f00f78a0 2021-11-20T05:37:19.865 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.867+0000 7f34d77fe700 1 -- 172.21.15.58:0/3122540023 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2950 (secure 0 0 0) 0x7f34e4023070 con 0x7f34f00f78a0 2021-11-20T05:37:19.866 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:19.866 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:07.267431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14622,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14622":{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:19.868 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.869+0000 7f34f5437700 1 -- 172.21.15.58:0/3122540023 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f34d805c300 msgr2=0x7f34d805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:19.869 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.869+0000 7f34f5437700 1 --2- 172.21.15.58:0/3122540023 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f34d805c300 0x7f34d805e7c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f34f0106ad0 tx=0x7f34e000b040).stop 2021-11-20T05:37:19.870 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.869+0000 7f34f5437700 1 -- 172.21.15.58:0/3122540023 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f34f00f78a0 msgr2=0x7f34f0107250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:19.870 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.869+0000 7f34f5437700 1 --2- 172.21.15.58:0/3122540023 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f34f00f78a0 0x7f34f0107250 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f34e4000c00 tx=0x7f34e400b040).stop 2021-11-20T05:37:19.871 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.869+0000 7f34f5437700 1 -- 172.21.15.58:0/3122540023 shutdown_connections 2021-11-20T05:37:19.871 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.869+0000 7f34f5437700 1 --2- 172.21.15.58:0/3122540023 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f34d805c300 0x7f34d805e7c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:19.871 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.869+0000 7f34f5437700 1 --2- 172.21.15.58:0/3122540023 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f34f00f6600 0x7f34f0106d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:19.872 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.869+0000 7f34f5437700 1 --2- 172.21.15.58:0/3122540023 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f34f00f78a0 0x7f34f0107250 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:19.872 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.869+0000 7f34f5437700 1 -- 172.21.15.58:0/3122540023 >> 172.21.15.58:0/3122540023 conn(0x7f34f00f1d40 msgr2=0x7f34f00fb2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:19.873 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.869+0000 7f34f5437700 1 -- 172.21.15.58:0/3122540023 shutdown_connections 2021-11-20T05:37:19.873 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:19.869+0000 7f34f5437700 1 -- 172.21.15.58:0/3122540023 wait complete. 2021-11-20T05:37:19.874 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 68 2021-11-20T05:37:19.879 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:07.267431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14622, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14622': {'gid': 14622, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 64, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/3191333321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 3191333321}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 3191333321}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:19.880 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14622:up:rejoin 2021-11-20T05:37:20.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:20 smithi187 conmon[36358]: audit 2021-11-20T05:37:19.868414+0000 mon.smithi058 (mon.0) 733 : 2021-11-20T05:37:20.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:20 smithi187 conmon[36358]: audit [DBG] from='client.? 172.21.15.58:0/3122540023' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:20.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:20 smithi058 conmon[33537]: audit 2021-11-20T05:37:19.868414+0000 mon.smithi058 2021-11-20T05:37:20.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:20 smithi058 conmon[33537]: (mon.0) 733 : audit [DBG] from='client.? 172.21.15.58:0/3122540023' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:20.881 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:21.133 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.133+0000 7f9459b58700 1 -- 172.21.15.58:0/161499866 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f94540f5300 msgr2=0x7f94540f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:21.134 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.133+0000 7f9459b58700 1 --2- 172.21.15.58:0/161499866 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f94540f5300 0x7f94540f7a80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f9448004660 tx=0x7f9448009b30).stop 2021-11-20T05:37:21.134 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.133+0000 7f9459b58700 1 -- 172.21.15.58:0/161499866 shutdown_connections 2021-11-20T05:37:21.134 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.133+0000 7f9459b58700 1 --2- 172.21.15.58:0/161499866 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f94540f5300 0x7f94540f7a80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:21.135 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.133+0000 7f9459b58700 1 --2- 172.21.15.58:0/161499866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f94540f4960 0x7f94540f4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:21.135 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.133+0000 7f9459b58700 1 -- 172.21.15.58:0/161499866 >> 172.21.15.58:0/161499866 conn(0x7f94540f0440 msgr2=0x7f94540f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:21.135 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.134+0000 7f9459b58700 1 -- 172.21.15.58:0/161499866 shutdown_connections 2021-11-20T05:37:21.135 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.134+0000 7f9459b58700 1 -- 172.21.15.58:0/161499866 wait complete. 2021-11-20T05:37:21.136 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.134+0000 7f9459b58700 1 Processor -- start 2021-11-20T05:37:21.136 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.134+0000 7f9459b58700 1 -- start start 2021-11-20T05:37:21.136 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.134+0000 7f9459b58700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f94540f4960 0x7f9454106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:21.136 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.134+0000 7f9459b58700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f94540f5300 0x7f94541073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:21.136 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.134+0000 7f9459b58700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9454107970 con 0x7f94540f4960 2021-11-20T05:37:21.137 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.134+0000 7f9459b58700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9454107ab0 con 0x7f94540f5300 2021-11-20T05:37:21.137 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.134+0000 7f94537fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f94540f4960 0x7f9454106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:21.137 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.135+0000 7f94537fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f94540f4960 0x7f9454106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41602/0 (socket says 172.21.15.58:41602) 2021-11-20T05:37:21.137 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.135+0000 7f94537fe700 1 -- 172.21.15.58:0/1513783094 learned_addr learned my addr 172.21.15.58:0/1513783094 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:21.138 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.135+0000 7f9452ffd700 1 --2- 172.21.15.58:0/1513783094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f94540f5300 0x7f94541073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:21.138 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.135+0000 7f94537fe700 1 -- 172.21.15.58:0/1513783094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f94540f5300 msgr2=0x7f94541073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:21.138 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.135+0000 7f94537fe700 1 --2- 172.21.15.58:0/1513783094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f94540f5300 0x7f94541073c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:21.138 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.135+0000 7f94537fe700 1 -- 172.21.15.58:0/1513783094 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9448005040 con 0x7f94540f4960 2021-11-20T05:37:21.138 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.135+0000 7f94537fe700 1 --2- 172.21.15.58:0/1513783094 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f94540f4960 0x7f9454106ed0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f944400c790 tx=0x7f9444007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:21.139 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.135+0000 7f943bfff700 1 -- 172.21.15.58:0/1513783094 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f944400e680 con 0x7f94540f4960 2021-11-20T05:37:21.139 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.135+0000 7f943bfff700 1 -- 172.21.15.58:0/1513783094 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f944400ec90 con 0x7f94540f4960 2021-11-20T05:37:21.139 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.135+0000 7f9459b58700 1 -- 172.21.15.58:0/1513783094 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945410c550 con 0x7f94540f4960 2021-11-20T05:37:21.139 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.135+0000 7f943bfff700 1 -- 172.21.15.58:0/1513783094 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9444012cb0 con 0x7f94540f4960 2021-11-20T05:37:21.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.135+0000 7f9459b58700 1 -- 172.21.15.58:0/1513783094 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945410cb10 con 0x7f94540f4960 2021-11-20T05:37:21.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.136+0000 7f9459b58700 1 -- 172.21.15.58:0/1513783094 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9454027a00 con 0x7f94540f4960 2021-11-20T05:37:21.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.136+0000 7f943bfff700 1 -- 172.21.15.58:0/1513783094 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9444010070 con 0x7f94540f4960 2021-11-20T05:37:21.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.137+0000 7f943bfff700 1 --2- 172.21.15.58:0/1513783094 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f943c05c2a0 0x7f943c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:21.140 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.137+0000 7f943bfff700 1 -- 172.21.15.58:0/1513783094 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f9444076240 con 0x7f94540f4960 2021-11-20T05:37:21.141 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.137+0000 7f9452ffd700 1 --2- 172.21.15.58:0/1513783094 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f943c05c2a0 0x7f943c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:21.141 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.137+0000 7f9452ffd700 1 --2- 172.21.15.58:0/1513783094 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f943c05c2a0 0x7f943c05e760 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f9448004e30 tx=0x7f9448009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:21.142 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.141+0000 7f943bfff700 1 -- 172.21.15.58:0/1513783094 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9444048cd0 con 0x7f94540f4960 2021-11-20T05:37:21.335 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.339+0000 7f9459b58700 1 -- 172.21.15.58:0/1513783094 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94540fee50 con 0x7f94540f4960 2021-11-20T05:37:21.336 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.339+0000 7f943bfff700 1 -- 172.21.15.58:0/1513783094 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2950 (secure 0 0 0) 0x7f944404c2f0 con 0x7f94540f4960 2021-11-20T05:37:21.337 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:21.337 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:07.267431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14622,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14622":{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:21.338 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.341+0000 7f9459b58700 1 -- 172.21.15.58:0/1513783094 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f943c05c2a0 msgr2=0x7f943c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:21.339 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.341+0000 7f9459b58700 1 --2- 172.21.15.58:0/1513783094 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f943c05c2a0 0x7f943c05e760 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f9448004e30 tx=0x7f9448009b30).stop 2021-11-20T05:37:21.339 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.341+0000 7f9459b58700 1 -- 172.21.15.58:0/1513783094 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f94540f4960 msgr2=0x7f9454106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:21.339 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.341+0000 7f9459b58700 1 --2- 172.21.15.58:0/1513783094 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f94540f4960 0x7f9454106ed0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f944400c790 tx=0x7f9444007d90).stop 2021-11-20T05:37:21.339 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.341+0000 7f9459b58700 1 -- 172.21.15.58:0/1513783094 shutdown_connections 2021-11-20T05:37:21.340 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.341+0000 7f9459b58700 1 --2- 172.21.15.58:0/1513783094 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f943c05c2a0 0x7f943c05e760 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:21.340 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.341+0000 7f9459b58700 1 --2- 172.21.15.58:0/1513783094 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f94540f4960 0x7f9454106ed0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:21.340 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.341+0000 7f9459b58700 1 --2- 172.21.15.58:0/1513783094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f94540f5300 0x7f94541073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:21.340 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.341+0000 7f9459b58700 1 -- 172.21.15.58:0/1513783094 >> 172.21.15.58:0/1513783094 conn(0x7f94540f0440 msgr2=0x7f94540fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:21.340 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.341+0000 7f9459b58700 1 -- 172.21.15.58:0/1513783094 shutdown_connections 2021-11-20T05:37:21.341 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:21.341+0000 7f9459b58700 1 -- 172.21.15.58:0/1513783094 wait complete. 2021-11-20T05:37:21.341 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 68 2021-11-20T05:37:21.350 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:07.267431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14622, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14622': {'gid': 14622, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 64, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/3191333321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 3191333321}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 3191333321}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:21.351 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14622:up:rejoin 2021-11-20T05:37:21.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:21 smithi187 conmon[36358]: cluster 2021-11-20T05:37:20.282290+0000 mgr.smithi058.zagdlk (mgr.14164) 289 : cluster [DBG] pgmap v261: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2021-11-20T05:37:21.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:21 smithi187 conmon[36358]: audit 2021-11-20T05:37:21.340280+0000 mon.smithi058 (mon.0) 734 : audit [DBG] from='client.? 172.21.15.58:0/1513783094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:21.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:21 smithi058 conmon[33537]: cluster 2021-11-20T05:37:20.282290+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:37:21.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:21 smithi058 conmon[33537]: 289 : cluster [DBG] pgmap v261: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2021-11-20T05:37:21.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:21 smithi058 conmon[33537]: audit 2021-11-20T05:37:21.340280+0000 mon.smithi058 (mon. 2021-11-20T05:37:21.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:21 smithi058 conmon[33537]: 0) 734 : audit [DBG] from='client.? 172.21.15.58:0/1513783094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:22.352 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:22.488 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.490+0000 7fce0a557700 1 -- 172.21.15.58:0/2431471728 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce040f6770 msgr2=0x7fce040f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:22.489 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.490+0000 7fce0a557700 1 --2- 172.21.15.58:0/2431471728 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce040f6770 0x7fce040f6bd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fcdf4004660 tx=0x7fcdf4009b30).stop 2021-11-20T05:37:22.489 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.490+0000 7fce0a557700 1 -- 172.21.15.58:0/2431471728 shutdown_connections 2021-11-20T05:37:22.489 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.490+0000 7fce0a557700 1 --2- 172.21.15.58:0/2431471728 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce040f6770 0x7fce040f6bd0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:22.489 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.490+0000 7fce0a557700 1 --2- 172.21.15.58:0/2431471728 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce040f54d0 0x7fce040f58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:22.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.490+0000 7fce0a557700 1 -- 172.21.15.58:0/2431471728 >> 172.21.15.58:0/2431471728 conn(0x7fce040f0c10 msgr2=0x7fce040f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:22.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.490+0000 7fce0a557700 1 -- 172.21.15.58:0/2431471728 shutdown_connections 2021-11-20T05:37:22.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.490+0000 7fce0a557700 1 -- 172.21.15.58:0/2431471728 wait complete. 2021-11-20T05:37:22.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.490+0000 7fce0a557700 1 Processor -- start 2021-11-20T05:37:22.491 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.490+0000 7fce0a557700 1 -- start start 2021-11-20T05:37:22.491 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.491+0000 7fce0a557700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce040f54d0 0x7fce0410bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:22.491 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.491+0000 7fce0a557700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce040f6770 0x7fce0410e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:22.491 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.491+0000 7fce0a557700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce040603f0 con 0x7fce040f54d0 2021-11-20T05:37:22.491 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.491+0000 7fce0a557700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce04060530 con 0x7fce040f6770 2021-11-20T05:37:22.492 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.491+0000 7fce09555700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce040f54d0 0x7fce0410bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:22.492 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.491+0000 7fce09555700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce040f54d0 0x7fce0410bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41612/0 (socket says 172.21.15.58:41612) 2021-11-20T05:37:22.492 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.491+0000 7fce09555700 1 -- 172.21.15.58:0/4101526833 learned_addr learned my addr 172.21.15.58:0/4101526833 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:22.492 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.491+0000 7fce09555700 1 -- 172.21.15.58:0/4101526833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce040f6770 msgr2=0x7fce0410e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:22.493 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.491+0000 7fce09555700 1 --2- 172.21.15.58:0/4101526833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce040f6770 0x7fce0410e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:37:22.493 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.491+0000 7fce09555700 1 -- 172.21.15.58:0/4101526833 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf4005040 con 0x7fce040f54d0 2021-11-20T05:37:22.493 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.491+0000 7fce09555700 1 --2- 172.21.15.58:0/4101526833 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce040f54d0 0x7fce0410bde0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fce0000e7c0 tx=0x7fce0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:22.493 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.491+0000 7fcdf9ffb700 1 -- 172.21.15.58:0/4101526833 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce000092e0 con 0x7fce040f54d0 2021-11-20T05:37:22.493 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.491+0000 7fcdf9ffb700 1 -- 172.21.15.58:0/4101526833 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce0000f040 con 0x7fce040f54d0 2021-11-20T05:37:22.494 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.491+0000 7fce0a557700 1 -- 172.21.15.58:0/4101526833 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0410e7d0 con 0x7fce040f54d0 2021-11-20T05:37:22.494 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.491+0000 7fcdf9ffb700 1 -- 172.21.15.58:0/4101526833 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce00014750 con 0x7fce040f54d0 2021-11-20T05:37:22.494 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.491+0000 7fce0a557700 1 -- 172.21.15.58:0/4101526833 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce0410ec50 con 0x7fce040f54d0 2021-11-20T05:37:22.495 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.492+0000 7fce0a557700 1 -- 172.21.15.58:0/4101526833 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce04045ad0 con 0x7fce040f54d0 2021-11-20T05:37:22.496 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.493+0000 7fcdf9ffb700 1 -- 172.21.15.58:0/4101526833 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fce0001b030 con 0x7fce040f54d0 2021-11-20T05:37:22.496 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.493+0000 7fcdf9ffb700 1 --2- 172.21.15.58:0/4101526833 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcdf005c300 0x7fcdf005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:22.496 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.493+0000 7fcdf9ffb700 1 -- 172.21.15.58:0/4101526833 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fce000762f0 con 0x7fce040f54d0 2021-11-20T05:37:22.497 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.493+0000 7fce08d54700 1 --2- 172.21.15.58:0/4101526833 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcdf005c300 0x7fcdf005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:22.497 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.494+0000 7fce08d54700 1 --2- 172.21.15.58:0/4101526833 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcdf005c300 0x7fcdf005e7c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fcdf4013530 tx=0x7fcdf400b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:22.497 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.497+0000 7fcdf9ffb700 1 -- 172.21.15.58:0/4101526833 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce00048c50 con 0x7fce040f54d0 2021-11-20T05:37:22.692 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.695+0000 7fce0a557700 1 -- 172.21.15.58:0/4101526833 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce04027a00 con 0x7fce040f54d0 2021-11-20T05:37:22.693 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.696+0000 7fcdf9ffb700 1 -- 172.21.15.58:0/4101526833 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2950 (secure 0 0 0) 0x7fce00019090 con 0x7fce040f54d0 2021-11-20T05:37:22.693 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:22.693 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:07.267431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14622,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14622":{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:22.695 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.697+0000 7fce0a557700 1 -- 172.21.15.58:0/4101526833 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcdf005c300 msgr2=0x7fcdf005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:22.695 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.697+0000 7fce0a557700 1 --2- 172.21.15.58:0/4101526833 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcdf005c300 0x7fcdf005e7c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fcdf4013530 tx=0x7fcdf400b040).stop 2021-11-20T05:37:22.695 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.697+0000 7fce0a557700 1 -- 172.21.15.58:0/4101526833 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce040f54d0 msgr2=0x7fce0410bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:22.696 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.697+0000 7fce0a557700 1 --2- 172.21.15.58:0/4101526833 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce040f54d0 0x7fce0410bde0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fce0000e7c0 tx=0x7fce0000b040).stop 2021-11-20T05:37:22.696 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.697+0000 7fce0a557700 1 -- 172.21.15.58:0/4101526833 shutdown_connections 2021-11-20T05:37:22.696 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.697+0000 7fce0a557700 1 --2- 172.21.15.58:0/4101526833 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcdf005c300 0x7fcdf005e7c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:22.696 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.697+0000 7fce0a557700 1 --2- 172.21.15.58:0/4101526833 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce040f54d0 0x7fce0410bde0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:22.697 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.697+0000 7fce0a557700 1 --2- 172.21.15.58:0/4101526833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce040f6770 0x7fce0410e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:37:22.697 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.697+0000 7fce0a557700 1 -- 172.21.15.58:0/4101526833 >> 172.21.15.58:0/4101526833 conn(0x7fce040f0c10 msgr2=0x7fce040fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:22.697 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.698+0000 7fce0a557700 1 -- 172.21.15.58:0/4101526833 shutdown_connections 2021-11-20T05:37:22.698 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:22.698+0000 7fce0a557700 1 -- 172.21.15.58:0/4101526833 wait complete. 2021-11-20T05:37:22.698 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 68 2021-11-20T05:37:22.706 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:07.267431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14622, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14622': {'gid': 14622, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 64, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/3191333321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 3191333321}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 3191333321}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:22.707 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14622:up:rejoin 2021-11-20T05:37:23.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:23 smithi187 conmon[36358]: cluster 2021-11-20T05:37:22.282830+0000 mgr.smithi058.zagdlk (mgr.14164) 290 : cluster [DBG] pgmap v262: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:37:23.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:23 smithi187 conmon[36358]: audit 2021-11-20T05:37:22.696815+0000 mon.smithi058 (mon.0) 735 : audit [DBG] from='client.? 172.21.15.58:0/4101526833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:23.708 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:23.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:23 smithi058 conmon[33537]: cluster 2021-11-20T05:37:22.282830+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:37:23.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:23 smithi058 conmon[33537]: 14164) 290 : cluster [DBG] pgmap v262: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:37:23.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:23 smithi058 conmon[33537]: audit 2021-11-20T05:37:22.696815+0000 mon.smithi058 (mon.0) 735 : audit [DBG] from='client.? 172.21.15.58:0/4101526833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:23.842 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.844+0000 7fd54b828700 1 -- 172.21.15.58:0/2430546438 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5440f5580 msgr2=0x7fd5440f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:23.842 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.844+0000 7fd54b828700 1 --2- 172.21.15.58:0/2430546438 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5440f5580 0x7fd5440f59a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fd540004660 tx=0x7fd540009b30).stop 2021-11-20T05:37:23.843 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.844+0000 7fd54b828700 1 -- 172.21.15.58:0/2430546438 shutdown_connections 2021-11-20T05:37:23.843 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.844+0000 7fd54b828700 1 --2- 172.21.15.58:0/2430546438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5440f6820 0x7fd5440f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:23.843 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.844+0000 7fd54b828700 1 --2- 172.21.15.58:0/2430546438 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5440f5580 0x7fd5440f59a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:23.843 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.844+0000 7fd54b828700 1 -- 172.21.15.58:0/2430546438 >> 172.21.15.58:0/2430546438 conn(0x7fd5440f0ce0 msgr2=0x7fd5440f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:23.843 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.844+0000 7fd54b828700 1 -- 172.21.15.58:0/2430546438 shutdown_connections 2021-11-20T05:37:23.844 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.844+0000 7fd54b828700 1 -- 172.21.15.58:0/2430546438 wait complete. 2021-11-20T05:37:23.844 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.844+0000 7fd54b828700 1 Processor -- start 2021-11-20T05:37:23.844 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.844+0000 7fd54b828700 1 -- start start 2021-11-20T05:37:23.845 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.844+0000 7fd54b828700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5440f5580 0x7fd544105db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:23.845 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.844+0000 7fd54b828700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5440f6820 0x7fd5441062a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:23.845 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.844+0000 7fd54b828700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd544106850 con 0x7fd5440f5580 2021-11-20T05:37:23.845 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.844+0000 7fd54b828700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd544106990 con 0x7fd5440f6820 2021-11-20T05:37:23.845 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.845+0000 7fd5495c4700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5440f5580 0x7fd544105db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:23.846 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.845+0000 7fd5495c4700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5440f5580 0x7fd544105db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41622/0 (socket says 172.21.15.58:41622) 2021-11-20T05:37:23.846 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.845+0000 7fd5495c4700 1 -- 172.21.15.58:0/1897881009 learned_addr learned my addr 172.21.15.58:0/1897881009 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:23.846 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.845+0000 7fd5495c4700 1 -- 172.21.15.58:0/1897881009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5440f6820 msgr2=0x7fd5441062a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:23.846 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.845+0000 7fd5495c4700 1 --2- 172.21.15.58:0/1897881009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5440f6820 0x7fd5441062a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:37:23.847 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.845+0000 7fd5495c4700 1 -- 172.21.15.58:0/1897881009 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd540005040 con 0x7fd5440f5580 2021-11-20T05:37:23.847 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.845+0000 7fd5495c4700 1 --2- 172.21.15.58:0/1897881009 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5440f5580 0x7fd544105db0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fd540009f50 tx=0x7fd540009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:23.847 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.845+0000 7fd539ffb700 1 -- 172.21.15.58:0/1897881009 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd540018070 con 0x7fd5440f5580 2021-11-20T05:37:23.847 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.845+0000 7fd539ffb700 1 -- 172.21.15.58:0/1897881009 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd540013410 con 0x7fd5440f5580 2021-11-20T05:37:23.848 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.845+0000 7fd54b828700 1 -- 172.21.15.58:0/1897881009 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd54410b3d0 con 0x7fd5440f5580 2021-11-20T05:37:23.848 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.845+0000 7fd539ffb700 1 -- 172.21.15.58:0/1897881009 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd54001c620 con 0x7fd5440f5580 2021-11-20T05:37:23.848 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.845+0000 7fd54b828700 1 -- 172.21.15.58:0/1897881009 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54410b990 con 0x7fd5440f5580 2021-11-20T05:37:23.850 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.846+0000 7fd54b828700 1 -- 172.21.15.58:0/1897881009 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd544027a00 con 0x7fd5440f5580 2021-11-20T05:37:23.850 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.847+0000 7fd539ffb700 1 -- 172.21.15.58:0/1897881009 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd54001d420 con 0x7fd5440f5580 2021-11-20T05:37:23.850 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.847+0000 7fd539ffb700 1 --2- 172.21.15.58:0/1897881009 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd53005c2a0 0x7fd53005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:23.851 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.847+0000 7fd539ffb700 1 -- 172.21.15.58:0/1897881009 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fd540078010 con 0x7fd5440f5580 2021-11-20T05:37:23.851 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.847+0000 7fd548dc3700 1 --2- 172.21.15.58:0/1897881009 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd53005c2a0 0x7fd53005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:23.851 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.848+0000 7fd548dc3700 1 --2- 172.21.15.58:0/1897881009 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd53005c2a0 0x7fd53005e760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fd534004870 tx=0x7fd5340079c0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:23.851 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:23.851+0000 7fd539ffb700 1 -- 172.21.15.58:0/1897881009 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd54004a9f0 con 0x7fd5440f5580 2021-11-20T05:37:24.045 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:24.049+0000 7fd54b828700 1 -- 172.21.15.58:0/1897881009 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5440fdd10 con 0x7fd5440f5580 2021-11-20T05:37:24.046 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:24.049+0000 7fd539ffb700 1 -- 172.21.15.58:0/1897881009 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2950 (secure 0 0 0) 0x7fd54004e010 con 0x7fd5440f5580 2021-11-20T05:37:24.047 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:24.047 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:07.267431+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14622,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14622":{"gid":14622,"name":"cephfs.smithi058.rxapzd","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.58:6827/3191333321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6826","nonce":3191333321},{"type":"v1","addr":"172.21.15.58:6827","nonce":3191333321}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:24.048 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:24.051+0000 7fd54b828700 1 -- 172.21.15.58:0/1897881009 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd53005c2a0 msgr2=0x7fd53005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:24.048 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:24.051+0000 7fd54b828700 1 --2- 172.21.15.58:0/1897881009 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd53005c2a0 0x7fd53005e760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fd534004870 tx=0x7fd5340079c0).stop 2021-11-20T05:37:24.049 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:24.051+0000 7fd54b828700 1 -- 172.21.15.58:0/1897881009 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5440f5580 msgr2=0x7fd544105db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:24.049 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:24.051+0000 7fd54b828700 1 --2- 172.21.15.58:0/1897881009 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5440f5580 0x7fd544105db0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fd540009f50 tx=0x7fd540009b30).stop 2021-11-20T05:37:24.049 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:24.051+0000 7fd54b828700 1 -- 172.21.15.58:0/1897881009 shutdown_connections 2021-11-20T05:37:24.049 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:24.051+0000 7fd54b828700 1 --2- 172.21.15.58:0/1897881009 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd53005c2a0 0x7fd53005e760 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:24.050 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:24.051+0000 7fd54b828700 1 --2- 172.21.15.58:0/1897881009 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd5440f5580 0x7fd544105db0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:24.050 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:24.051+0000 7fd54b828700 1 --2- 172.21.15.58:0/1897881009 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5440f6820 0x7fd5441062a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:37:24.050 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:24.051+0000 7fd54b828700 1 -- 172.21.15.58:0/1897881009 >> 172.21.15.58:0/1897881009 conn(0x7fd5440f0ce0 msgr2=0x7fd5440fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:24.050 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:24.051+0000 7fd54b828700 1 -- 172.21.15.58:0/1897881009 shutdown_connections 2021-11-20T05:37:24.051 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:24.051+0000 7fd54b828700 1 -- 172.21.15.58:0/1897881009 wait complete. 2021-11-20T05:37:24.051 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 68 2021-11-20T05:37:24.060 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:07.267431+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14622, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14622': {'gid': 14622, 'name': 'cephfs.smithi058.rxapzd', 'rank': 0, 'incarnation': 64, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.58:6827/3191333321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6826', 'nonce': 3191333321}, {'type': 'v1', 'addr': '172.21.15.58:6827', 'nonce': 3191333321}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:24.060 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14622:up:rejoin 2021-11-20T05:37:24.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:24 smithi187 conmon[36358]: audit 2021-11-20T05:37:24.050231+0000 mon.smithi058 (mon.0) 736 : audit [DBG] from='client.? 172.21.15.58:0/1897881009' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:24.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:24 smithi187 conmon[36358]: cluster 2021-11-20T05:37:24.259271+0000 mon.smithi058 (mon.0) 737 : cluster [WRN] Replacing daemon mds.cephfs.smithi058.rxapzd as rank 0 with standby daemon mds.cephfs.smithi187.fikisa 2021-11-20T05:37:24.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:24 smithi187 conmon[36358]: cluster 2021-11-20T05:37:24.259314+0000 mon.smithi058 (mon.0) 738 : cluster [INF] MDS daemon mds.cephfs.smithi058.rxapzd is removed because it is dead or otherwise unavailable. 2021-11-20T05:37:24.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:24 smithi187 conmon[36358]: cluster 2021-11-20T05:37:24.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:24 smithi187 conmon[36358]: 2021-11-20T05:37:24.263214+0000 mon.smithi058 (mon.0) 739 : cluster [DBG] 2021-11-20T05:37:24.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:24 smithi187 conmon[36358]: osdmap e53: 6 total, 6 up, 6 in 2021-11-20T05:37:24.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:24 smithi187 conmon[36358]: cluster 2021-11-20T05:37:24.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:24 smithi187 conmon[36358]: 2021-11-20T05:37:24 2021-11-20T05:37:24.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:24 smithi187 conmon[36358]: .266023+0000 mon.smithi058 (mon.0) 2021-11-20T05:37:24.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:24 smithi187 conmon[36358]: 740 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:24.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:24 smithi058 conmon[33537]: audit 2021-11-20T05:37:24.050231+0000 mon.smithi058 (mon.0) 736 : audit [DBG] from='client.? 172.21.15.58:0/1897881009' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:24.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:24 smithi058 conmon[33537]: cluster 2021-11-20T05:37:24.259271+0000 mon.smithi058 (mon.0) 737 : cluster [WRN] Replacing daemon mds.cephfs.smithi058.rxapzd as rank 0 with standby daemon mds.cephfs.smithi187.fikisa 2021-11-20T05:37:24.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:24 smithi058 conmon[33537]: cluster 2021-11-20T05:37:24.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:24 smithi058 conmon[33537]: 2021-11-20T05:37:24.259314+0000 2021-11-20T05:37:24.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:24 smithi058 conmon[33537]: mon.smithi058 (mon.0) 738 : 2021-11-20T05:37:24.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:24 smithi058 conmon[33537]: cluster [INF] MDS daemon mds.cephfs.smithi058.rxapzd is removed because it is dead or otherwise unavailable. 2021-11-20T05:37:24.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:24 smithi058 conmon[33537]: cluster 2021-11-20T05:37:24.263214+0000 2021-11-20T05:37:24.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:24 smithi058 conmon[33537]: mon.smithi058 (mon.0) 739 : cluster 2021-11-20T05:37:24.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:24 smithi058 conmon[33537]: [DBG] osdmap e53: 6 total, 6 up, 6 in 2021-11-20T05:37:24.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:24 smithi058 conmon[33537]: cluster 2021-11-20T05 2021-11-20T05:37:24.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:24 smithi058 conmon[33537]: :37:24.266023+0000 mon.smithi058 (mon. 2021-11-20T05:37:24.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:24 smithi058 conmon[33537]: 0) 740 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:25.061 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:25.198 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.197+0000 7f6bc6918700 1 -- 172.21.15.58:0/3104118752 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6bc00f6840 msgr2=0x7f6bc00f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:25.198 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.197+0000 7f6bc6918700 1 --2- 172.21.15.58:0/3104118752 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6bc00f6840 0x7f6bc00f6ca0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f6bb4004660 tx=0x7f6bb4009b30).stop 2021-11-20T05:37:25.198 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.197+0000 7f6bc6918700 1 -- 172.21.15.58:0/3104118752 shutdown_connections 2021-11-20T05:37:25.199 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.197+0000 7f6bc6918700 1 --2- 172.21.15.58:0/3104118752 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6bc00f6840 0x7f6bc00f6ca0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:25.199 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.197+0000 7f6bc6918700 1 --2- 172.21.15.58:0/3104118752 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6bc00f55a0 0x7f6bc00f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:37:25.199 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.197+0000 7f6bc6918700 1 -- 172.21.15.58:0/3104118752 >> 172.21.15.58:0/3104118752 conn(0x7f6bc00f0ce0 msgr2=0x7f6bc00f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:25.199 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.197+0000 7f6bc6918700 1 -- 172.21.15.58:0/3104118752 shutdown_connections 2021-11-20T05:37:25.200 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.197+0000 7f6bc6918700 1 -- 172.21.15.58:0/3104118752 wait complete. 2021-11-20T05:37:25.200 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.197+0000 7f6bc6918700 1 Processor -- start 2021-11-20T05:37:25.200 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.198+0000 7f6bc6918700 1 -- start start 2021-11-20T05:37:25.200 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.198+0000 7f6bc6918700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6bc00f55a0 0x7f6bc0106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:25.201 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.198+0000 7f6bc6918700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6bc00f6840 0x7f6bc01073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:25.201 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.198+0000 7f6bc6918700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc01079f0 con 0x7f6bc00f6840 2021-11-20T05:37:25.201 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.198+0000 7f6bc6918700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc0107b30 con 0x7f6bc00f55a0 2021-11-20T05:37:25.201 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.198+0000 7f6bbf7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6bc00f6840 0x7f6bc01073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:25.201 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.198+0000 7f6bbf7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6bc00f6840 0x7f6bc01073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41634/0 (socket says 172.21.15.58:41634) 2021-11-20T05:37:25.202 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.198+0000 7f6bbf7fe700 1 -- 172.21.15.58:0/1389248600 learned_addr learned my addr 172.21.15.58:0/1389248600 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:25.202 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.198+0000 7f6bbffff700 1 --2- 172.21.15.58:0/1389248600 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6bc00f55a0 0x7f6bc0106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:25.202 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.198+0000 7f6bbf7fe700 1 -- 172.21.15.58:0/1389248600 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6bc00f55a0 msgr2=0x7f6bc0106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:25.202 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.198+0000 7f6bbf7fe700 1 --2- 172.21.15.58:0/1389248600 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6bc00f55a0 0x7f6bc0106ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:25.202 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.198+0000 7f6bbf7fe700 1 -- 172.21.15.58:0/1389248600 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb4005040 con 0x7f6bc00f6840 2021-11-20T05:37:25.203 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.198+0000 7f6bbf7fe700 1 --2- 172.21.15.58:0/1389248600 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6bc00f6840 0x7f6bc01073b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f6bb4009830 tx=0x7f6bb4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:25.203 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.199+0000 7f6bbcff9700 1 -- 172.21.15.58:0/1389248600 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bb4018070 con 0x7f6bc00f6840 2021-11-20T05:37:25.203 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.199+0000 7f6bc6918700 1 -- 172.21.15.58:0/1389248600 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc010c570 con 0x7f6bc00f6840 2021-11-20T05:37:25.203 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.199+0000 7f6bbcff9700 1 -- 172.21.15.58:0/1389248600 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bb4013410 con 0x7f6bc00f6840 2021-11-20T05:37:25.204 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.199+0000 7f6bbcff9700 1 -- 172.21.15.58:0/1389248600 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bb401c620 con 0x7f6bc00f6840 2021-11-20T05:37:25.204 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.199+0000 7f6bc6918700 1 -- 172.21.15.58:0/1389248600 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc010cad0 con 0x7f6bc00f6840 2021-11-20T05:37:25.204 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.200+0000 7f6bc6918700 1 -- 172.21.15.58:0/1389248600 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc00fddd0 con 0x7f6bc00f6840 2021-11-20T05:37:25.204 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.200+0000 7f6bbcff9700 1 -- 172.21.15.58:0/1389248600 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6bb401d420 con 0x7f6bc00f6840 2021-11-20T05:37:25.204 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.200+0000 7f6bbcff9700 1 --2- 172.21.15.58:0/1389248600 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6ba805c2a0 0x7f6ba805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:25.205 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.201+0000 7f6bbcff9700 1 -- 172.21.15.58:0/1389248600 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6bb40780e0 con 0x7f6bc00f6840 2021-11-20T05:37:25.205 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.201+0000 7f6bbffff700 1 --2- 172.21.15.58:0/1389248600 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6ba805c2a0 0x7f6ba805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:25.205 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.201+0000 7f6bbffff700 1 --2- 172.21.15.58:0/1389248600 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6ba805c2a0 0x7f6ba805e760 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f6bb0006fd0 tx=0x7f6bb0008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:25.206 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.205+0000 7f6bbcff9700 1 -- 172.21.15.58:0/1389248600 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6bb404a9f0 con 0x7f6bc00f6840 2021-11-20T05:37:25.399 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.403+0000 7f6bc6918700 1 -- 172.21.15.58:0/1389248600 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bc010d3e0 con 0x7f6bc00f6840 2021-11-20T05:37:25.400 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.403+0000 7f6bbcff9700 1 -- 172.21.15.58:0/1389248600 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+2577 (secure 0 0 0) 0x7f6bb404e010 con 0x7f6bc00f6840 2021-11-20T05:37:25.401 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:25.401 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:25.264476+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24371,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24371":{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":69,"state":"up:resolve","state_seq":16,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:25.402 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.405+0000 7f6bc6918700 1 -- 172.21.15.58:0/1389248600 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6ba805c2a0 msgr2=0x7f6ba805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:25.402 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.405+0000 7f6bc6918700 1 --2- 172.21.15.58:0/1389248600 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6ba805c2a0 0x7f6ba805e760 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f6bb0006fd0 tx=0x7f6bb0008040).stop 2021-11-20T05:37:25.403 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.405+0000 7f6bc6918700 1 -- 172.21.15.58:0/1389248600 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6bc00f6840 msgr2=0x7f6bc01073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:25.403 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.405+0000 7f6bc6918700 1 --2- 172.21.15.58:0/1389248600 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6bc00f6840 0x7f6bc01073b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f6bb4009830 tx=0x7f6bb4009b30).stop 2021-11-20T05:37:25.403 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.405+0000 7f6bc6918700 1 -- 172.21.15.58:0/1389248600 shutdown_connections 2021-11-20T05:37:25.403 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.405+0000 7f6bc6918700 1 --2- 172.21.15.58:0/1389248600 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f6ba805c2a0 0x7f6ba805e760 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:25.404 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.405+0000 7f6bc6918700 1 --2- 172.21.15.58:0/1389248600 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6bc00f55a0 0x7f6bc0106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:25.404 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.405+0000 7f6bc6918700 1 --2- 172.21.15.58:0/1389248600 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f6bc00f6840 0x7f6bc01073b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:25.404 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.405+0000 7f6bc6918700 1 -- 172.21.15.58:0/1389248600 >> 172.21.15.58:0/1389248600 conn(0x7f6bc00f0ce0 msgr2=0x7f6bc005fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:25.404 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.405+0000 7f6bc6918700 1 -- 172.21.15.58:0/1389248600 shutdown_connections 2021-11-20T05:37:25.405 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:25.405+0000 7f6bc6918700 1 -- 172.21.15.58:0/1389248600 wait complete. 2021-11-20T05:37:25.405 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 70 2021-11-20T05:37:25.414 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:25.264476+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24371, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24371': {'gid': 24371, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 69, 'state': 'up:resolve', 'state_seq': 16, 'addr': '172.21.15.187:6827/2249219455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 2249219455}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 2249219455}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:25.414 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24371:up:resolve 2021-11-20T05:37:25.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:25 smithi187 conmon[36358]: cluster 2021-11-20T05:37:24.283374+0000 mgr.smithi058.zagdlk (mgr.14164) 291 : cluster [DBG] pgmap v264: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:37:25.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:25 smithi187 conmon[36358]: cluster 2021-11-20T05:37:25.270421+0000 mon.smithi058 (mon.0) 741 : cluster [DBG] mds.? [v2:172.21.15.187:6826/2249219455,v1:172.21.15.187:6827/2249219455] up:resolve 2021-11-20T05:37:25.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:25 smithi187 conmon[36358]: cluster 2021-11-20T05:37:25.270519+0000 mon.smithi058 (mon.0) 742 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:25.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:25 smithi058 conmon[33537]: cluster 2021-11-20T05:37:24.283374+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:37:25.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:25 smithi058 conmon[33537]: mgr.14164) 291 : cluster [DBG] pgmap v264: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:37:25.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:25 smithi058 conmon[33537]: cluster 2021-11-20T05:37:25.270421+0000 mon.smithi058 (mon.0) 741 : 2021-11-20T05:37:25.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:25 smithi058 conmon[33537]: cluster [DBG] mds.? [v2:172.21.15.187:6826/2249219455,v1:172.21.15.187:6827/2249219455] up:resolve 2021-11-20T05:37:25.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:25 smithi058 conmon[33537]: cluster 2021-11-20T05:37:25.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:25 smithi058 conmon[33537]: 2021-11-20T05:37:25 2021-11-20T05:37:25.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:25 smithi058 conmon[33537]: .270519+0000 mon.smithi058 (mon.0) 742 : 2021-11-20T05:37:25.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:25 smithi058 conmon[33537]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:26.415 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:26.547 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.548+0000 7f1a72900700 1 -- 172.21.15.58:0/4254766414 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a6c0f6840 msgr2=0x7f1a6c0f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:26.547 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.548+0000 7f1a72900700 1 --2- 172.21.15.58:0/4254766414 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a6c0f6840 0x7f1a6c0f6ca0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f1a54004660 tx=0x7f1a54009b30).stop 2021-11-20T05:37:26.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.548+0000 7f1a72900700 1 -- 172.21.15.58:0/4254766414 shutdown_connections 2021-11-20T05:37:26.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.548+0000 7f1a72900700 1 --2- 172.21.15.58:0/4254766414 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a6c0f6840 0x7f1a6c0f6ca0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:26.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.548+0000 7f1a72900700 1 --2- 172.21.15.58:0/4254766414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a6c0f55a0 0x7f1a6c0f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:26.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.548+0000 7f1a72900700 1 -- 172.21.15.58:0/4254766414 >> 172.21.15.58:0/4254766414 conn(0x7f1a6c0f0ce0 msgr2=0x7f1a6c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:26.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.548+0000 7f1a72900700 1 -- 172.21.15.58:0/4254766414 shutdown_connections 2021-11-20T05:37:26.549 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.548+0000 7f1a72900700 1 -- 172.21.15.58:0/4254766414 wait complete. 2021-11-20T05:37:26.549 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.549+0000 7f1a72900700 1 Processor -- start 2021-11-20T05:37:26.549 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.549+0000 7f1a72900700 1 -- start start 2021-11-20T05:37:26.549 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.549+0000 7f1a72900700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a6c0f55a0 0x7f1a6c05ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:26.550 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.549+0000 7f1a72900700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a6c0f6840 0x7f1a6c10bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:26.550 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.549+0000 7f1a72900700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c10c180 con 0x7f1a6c0f55a0 2021-11-20T05:37:26.550 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.549+0000 7f1a72900700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c10c2c0 con 0x7f1a6c0f6840 2021-11-20T05:37:26.550 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.549+0000 7f1a6bfff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a6c0f55a0 0x7f1a6c05ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:26.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.549+0000 7f1a6bfff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a6c0f55a0 0x7f1a6c05ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41644/0 (socket says 172.21.15.58:41644) 2021-11-20T05:37:26.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.549+0000 7f1a6bfff700 1 -- 172.21.15.58:0/1677176784 learned_addr learned my addr 172.21.15.58:0/1677176784 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:26.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.549+0000 7f1a6bfff700 1 -- 172.21.15.58:0/1677176784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a6c0f6840 msgr2=0x7f1a6c10bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:26.551 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.549+0000 7f1a6bfff700 1 --2- 172.21.15.58:0/1677176784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a6c0f6840 0x7f1a6c10bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:37:26.552 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.549+0000 7f1a6bfff700 1 -- 172.21.15.58:0/1677176784 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a54005040 con 0x7f1a6c0f55a0 2021-11-20T05:37:26.552 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.549+0000 7f1a6bfff700 1 --2- 172.21.15.58:0/1677176784 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a6c0f55a0 0x7f1a6c05ff20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f1a5c00c7c0 tx=0x7f1a5c007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:26.552 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.550+0000 7f1a697fa700 1 -- 172.21.15.58:0/1677176784 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a5c00e680 con 0x7f1a6c0f55a0 2021-11-20T05:37:26.552 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.550+0000 7f1a697fa700 1 -- 172.21.15.58:0/1677176784 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a5c00ec90 con 0x7f1a6c0f55a0 2021-11-20T05:37:26.553 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.550+0000 7f1a72900700 1 -- 172.21.15.58:0/1677176784 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6c10c520 con 0x7f1a6c0f55a0 2021-11-20T05:37:26.553 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.550+0000 7f1a697fa700 1 -- 172.21.15.58:0/1677176784 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a5c012ce0 con 0x7f1a6c0f55a0 2021-11-20T05:37:26.553 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.550+0000 7f1a72900700 1 -- 172.21.15.58:0/1677176784 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6c10cae0 con 0x7f1a6c0f55a0 2021-11-20T05:37:26.553 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.550+0000 7f1a72900700 1 -- 172.21.15.58:0/1677176784 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6c027a00 con 0x7f1a6c0f55a0 2021-11-20T05:37:26.553 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.551+0000 7f1a697fa700 1 -- 172.21.15.58:0/1677176784 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1a5c010070 con 0x7f1a6c0f55a0 2021-11-20T05:37:26.555 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.551+0000 7f1a697fa700 1 --2- 172.21.15.58:0/1677176784 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1a5805c2a0 0x7f1a5805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:26.555 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.551+0000 7f1a697fa700 1 -- 172.21.15.58:0/1677176784 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1a5c076340 con 0x7f1a6c0f55a0 2021-11-20T05:37:26.555 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.551+0000 7f1a63fff700 1 --2- 172.21.15.58:0/1677176784 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1a5805c2a0 0x7f1a5805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:26.555 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.552+0000 7f1a63fff700 1 --2- 172.21.15.58:0/1677176784 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1a5805c2a0 0x7f1a5805e760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f1a54000c00 tx=0x7f1a54009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:26.556 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.555+0000 7f1a697fa700 1 -- 172.21.15.58:0/1677176784 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a5c048c50 con 0x7f1a6c0f55a0 2021-11-20T05:37:26.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:26 smithi187 conmon[36358]: audit 2021-11-20T05:37:25.404297+0000 mon.smithi058 (mon.0) 743 : audit [DBG] from='client.? 172.21.15.58:0/1389248600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:26.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:26 smithi187 conmon[36358]: cluster 2021-11-20T05:37:26.273250+0000 mon.smithi058 (mon.0) 744 : cluster [DBG] mds.? [v2:172.21.15.187:6826/2249219455,v1:172.21.15.187:6827/2249219455] up:reconnect 2021-11-20T05:37:26.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:26 smithi187 conmon[36358]: cluster 2021-11-20T05:37:26.273322+0000 mon.smithi058 (mon.0) 745 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:26.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:26 smithi058 conmon[33537]: audit 2021-11-20T05:37:25.404297+0000 mon.smithi058 (mon.0 2021-11-20T05:37:26.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:26 smithi058 conmon[33537]: ) 743 : audit [DBG] from='client.? 172.21.15.58:0/1389248600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:26.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:26 smithi058 conmon[33537]: cluster 2021-11-20T05:37:26.273250+0000 mon.smithi058 (mon.0) 744 : cluster [DBG] mds.? [v2:172.21.15.187:6826/2249219455,v1:172.21.15.187:6827/2249219455] up:reconnect 2021-11-20T05:37:26.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:26 smithi058 conmon[33537]: cluster 2021-11-20T05:37:26.273322 2021-11-20T05:37:26.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:26 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 745 : cluster [DBG] 2021-11-20T05:37:26.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:26 smithi058 conmon[33537]: fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:26.746 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.750+0000 7f1a72900700 1 -- 172.21.15.58:0/1677176784 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a6c0fdd30 con 0x7f1a6c0f55a0 2021-11-20T05:37:26.747 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.751+0000 7f1a697fa700 1 -- 172.21.15.58:0/1677176784 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2579 (secure 0 0 0) 0x7f1a5c04c270 con 0x7f1a6c0f55a0 2021-11-20T05:37:26.748 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:26.748 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:26.268673+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24371,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24371":{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":69,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:26.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.752+0000 7f1a72900700 1 -- 172.21.15.58:0/1677176784 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1a5805c2a0 msgr2=0x7f1a5805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:26.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.752+0000 7f1a72900700 1 --2- 172.21.15.58:0/1677176784 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1a5805c2a0 0x7f1a5805e760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f1a54000c00 tx=0x7f1a54009b30).stop 2021-11-20T05:37:26.750 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.752+0000 7f1a72900700 1 -- 172.21.15.58:0/1677176784 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a6c0f55a0 msgr2=0x7f1a6c05ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:26.750 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.752+0000 7f1a72900700 1 --2- 172.21.15.58:0/1677176784 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a6c0f55a0 0x7f1a6c05ff20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f1a5c00c7c0 tx=0x7f1a5c007d90).stop 2021-11-20T05:37:26.750 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.752+0000 7f1a72900700 1 -- 172.21.15.58:0/1677176784 shutdown_connections 2021-11-20T05:37:26.751 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.752+0000 7f1a72900700 1 --2- 172.21.15.58:0/1677176784 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1a5805c2a0 0x7f1a5805e760 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:26.751 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.752+0000 7f1a72900700 1 --2- 172.21.15.58:0/1677176784 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a6c0f55a0 0x7f1a6c05ff20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:26.751 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.752+0000 7f1a72900700 1 --2- 172.21.15.58:0/1677176784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a6c0f6840 0x7f1a6c10bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:37:26.751 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.752+0000 7f1a72900700 1 -- 172.21.15.58:0/1677176784 >> 172.21.15.58:0/1677176784 conn(0x7f1a6c0f0ce0 msgr2=0x7f1a6c0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:26.752 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.752+0000 7f1a72900700 1 -- 172.21.15.58:0/1677176784 shutdown_connections 2021-11-20T05:37:26.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:26.752+0000 7f1a72900700 1 -- 172.21.15.58:0/1677176784 wait complete. 2021-11-20T05:37:26.753 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 71 2021-11-20T05:37:26.761 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:26.268673+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24371, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24371': {'gid': 24371, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 69, 'state': 'up:reconnect', 'state_seq': 17, 'addr': '172.21.15.187:6827/2249219455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 2249219455}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 2249219455}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:26.761 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24371:up:reconnect 2021-11-20T05:37:27.667 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:27 smithi187 conmon[36358]: cluster 2021-11-20T05:37:26.284118+0000 mgr.smithi058.zagdlk (mgr.14164) 292 : cluster [DBG] pgmap v265: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:37:27.667 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:27 smithi187 conmon[36358]: cluster 2021-11-20T05:37:26.292657+0000 mds.cephfs.smithi187.fikisa (mds.0) 1 : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0219996 2021-11-20T05:37:27.667 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:27 smithi187 conmon[36358]: audit 2021-11-20T05:37:26.751600 2021-11-20T05:37:27.668 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:27 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 746 : audit [DBG] from='client.? 172.21.15.58:0/1677176784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:27.668 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:27 smithi187 conmon[36358]: 2021-11-20T05:37:27.668 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:27 smithi187 conmon[36358]: cluster 2021-11-20T05: 2021-11-20T05:37:27.669 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:27 smithi187 conmon[36358]: 37:27.275942+0000 mon.smithi058 (mon.0) 747 : cluster [DBG] 2021-11-20T05:37:27.669 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:27 smithi187 conmon[36358]: mds.? [v2:172.21.15.187:6826/2249219455,v1:172.21.15.187:6827/2249219455] up:rejoin 2021-11-20T05:37:27.669 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:27 smithi187 conmon[36358]: cluster 2021-11-20T05:37:27.669 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:27 smithi187 conmon[36358]: 2021-11-20T05:37:27.276016+0000 mon.smithi058 2021-11-20T05:37:27.670 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:27 smithi187 conmon[36358]: (mon.0) 748 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:27.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:27 smithi058 conmon[33537]: cluster 2021-11-20T05:37:26.284118+0000 mgr.smithi058.zagdlk (mgr.14164) 292 : cluster 2021-11-20T05:37:27.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:27 smithi058 conmon[33537]: [DBG] pgmap v265: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:37:27.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:27 smithi058 conmon[33537]: cluster 2021-11-20T05:37:26.292657+0000 mds.cephfs.smithi187.fikisa (mds.0) 1 : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0219996 2021-11-20T05:37:27.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:27 smithi058 conmon[33537]: audit 2021-11-20T05: 2021-11-20T05:37:27.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:27 smithi058 conmon[33537]: 37:26.751600+0000 mon.smithi058 (mon 2021-11-20T05:37:27.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:27 smithi058 conmon[33537]: .0) 746 : audit [DBG] from='client.? 172.21.15.58:0/1677176784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:27.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:27 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:37:27.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:27 smithi058 conmon[33537]: -11-20T05:37:27.275942 2021-11-20T05:37:27.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:27 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 747 : cluster 2021-11-20T05:37:27.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:27 smithi058 conmon[33537]: [DBG] mds.? [v2:172.21.15.187:6826/2249219455,v1:172.21.15.187:6827/2249219455] up:rejoin 2021-11-20T05:37:27.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:27 smithi058 conmon[33537]: cluster 2021-11-20T05: 2021-11-20T05:37:27.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:27 smithi058 conmon[33537]: 37:27.276016+0000 mon.smithi058 (mon.0 2021-11-20T05:37:27.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:27 smithi058 conmon[33537]: ) 748 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:27.762 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:27.897 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.898+0000 7f22ca468700 1 -- 172.21.15.58:0/1796195552 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f22c40600e0 msgr2=0x7f22c40f91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:27.898 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.898+0000 7f22ca468700 1 --2- 172.21.15.58:0/1796195552 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f22c40600e0 0x7f22c40f91a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f22bc0099e0 tx=0x7f22bc004d10).stop 2021-11-20T05:37:27.898 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.899+0000 7f22ca468700 1 -- 172.21.15.58:0/1796195552 shutdown_connections 2021-11-20T05:37:27.898 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.899+0000 7f22ca468700 1 --2- 172.21.15.58:0/1796195552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22c40f9720 0x7f22c40fbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:27.899 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.899+0000 7f22ca468700 1 --2- 172.21.15.58:0/1796195552 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f22c40600e0 0x7f22c40f91a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:27.899 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.899+0000 7f22ca468700 1 -- 172.21.15.58:0/1796195552 >> 172.21.15.58:0/1796195552 conn(0x7f22c40f0ba0 msgr2=0x7f22c40f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:27.899 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.899+0000 7f22ca468700 1 -- 172.21.15.58:0/1796195552 shutdown_connections 2021-11-20T05:37:27.899 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.899+0000 7f22ca468700 1 -- 172.21.15.58:0/1796195552 wait complete. 2021-11-20T05:37:27.900 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.899+0000 7f22ca468700 1 Processor -- start 2021-11-20T05:37:27.900 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.899+0000 7f22ca468700 1 -- start start 2021-11-20T05:37:27.900 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.899+0000 7f22ca468700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f22c40600e0 0x7f22c4106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:27.900 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.899+0000 7f22ca468700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22c40f9720 0x7f22c4107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:27.900 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.899+0000 7f22ca468700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c4107930 con 0x7f22c40600e0 2021-11-20T05:37:27.901 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.899+0000 7f22ca468700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c4107a70 con 0x7f22c40f9720 2021-11-20T05:37:27.901 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.899+0000 7f22c9466700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f22c40600e0 0x7f22c4106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:27.901 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.899+0000 7f22c9466700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f22c40600e0 0x7f22c4106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41660/0 (socket says 172.21.15.58:41660) 2021-11-20T05:37:27.901 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.899+0000 7f22c9466700 1 -- 172.21.15.58:0/1920844496 learned_addr learned my addr 172.21.15.58:0/1920844496 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:27.902 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.900+0000 7f22c8c65700 1 --2- 172.21.15.58:0/1920844496 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22c40f9720 0x7f22c4107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:27.902 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.900+0000 7f22c9466700 1 -- 172.21.15.58:0/1920844496 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22c40f9720 msgr2=0x7f22c4107380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:27.902 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.900+0000 7f22c9466700 1 --2- 172.21.15.58:0/1920844496 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22c40f9720 0x7f22c4107380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:27.902 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.900+0000 7f22c9466700 1 -- 172.21.15.58:0/1920844496 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc009640 con 0x7f22c40600e0 2021-11-20T05:37:27.903 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.900+0000 7f22c8c65700 1 --2- 172.21.15.58:0/1920844496 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22c40f9720 0x7f22c4107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:37:27.903 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.900+0000 7f22c9466700 1 --2- 172.21.15.58:0/1920844496 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f22c40600e0 0x7f22c4106e90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f22bc004b70 tx=0x7f22bc004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:27.903 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.900+0000 7f22c1ffb700 1 -- 172.21.15.58:0/1920844496 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22bc00e030 con 0x7f22c40600e0 2021-11-20T05:37:27.903 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.900+0000 7f22c1ffb700 1 -- 172.21.15.58:0/1920844496 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f22bc01d8d0 con 0x7f22c40600e0 2021-11-20T05:37:27.903 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.900+0000 7f22ca468700 1 -- 172.21.15.58:0/1920844496 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c41083b0 con 0x7f22c40600e0 2021-11-20T05:37:27.904 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.900+0000 7f22c1ffb700 1 -- 172.21.15.58:0/1920844496 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22bc018070 con 0x7f22c40600e0 2021-11-20T05:37:27.904 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.900+0000 7f22ca468700 1 -- 172.21.15.58:0/1920844496 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c4108890 con 0x7f22c40600e0 2021-11-20T05:37:27.904 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.901+0000 7f22ca468700 1 -- 172.21.15.58:0/1920844496 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c4045ad0 con 0x7f22c40600e0 2021-11-20T05:37:27.905 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.902+0000 7f22c1ffb700 1 -- 172.21.15.58:0/1920844496 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f22bc01da40 con 0x7f22c40600e0 2021-11-20T05:37:27.906 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.902+0000 7f22c1ffb700 1 --2- 172.21.15.58:0/1920844496 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f22ac05c2f0 0x7f22ac05e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:27.906 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.902+0000 7f22c1ffb700 1 -- 172.21.15.58:0/1920844496 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f22bc0781b0 con 0x7f22c40600e0 2021-11-20T05:37:27.906 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.902+0000 7f22c8c65700 1 --2- 172.21.15.58:0/1920844496 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f22ac05c2f0 0x7f22ac05e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:27.906 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.903+0000 7f22c8c65700 1 --2- 172.21.15.58:0/1920844496 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f22ac05c2f0 0x7f22ac05e7b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f22b8009fd0 tx=0x7f22b800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:27.907 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:27.906+0000 7f22c1ffb700 1 -- 172.21.15.58:0/1920844496 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f22bc04aae0 con 0x7f22c40600e0 2021-11-20T05:37:28.103 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:28.107+0000 7f22ca468700 1 -- 172.21.15.58:0/1920844496 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22c40fede0 con 0x7f22c40600e0 2021-11-20T05:37:28.104 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:28.107+0000 7f22c1ffb700 1 -- 172.21.15.58:0/1920844496 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2576 (secure 0 0 0) 0x7f22bc04e100 con 0x7f22c40600e0 2021-11-20T05:37:28.104 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:28.104 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:27.270544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24371,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24371":{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:28.105 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:28.109+0000 7f22ca468700 1 -- 172.21.15.58:0/1920844496 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f22ac05c2f0 msgr2=0x7f22ac05e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:28.106 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:28.109+0000 7f22ca468700 1 --2- 172.21.15.58:0/1920844496 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f22ac05c2f0 0x7f22ac05e7b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f22b8009fd0 tx=0x7f22b800b040).stop 2021-11-20T05:37:28.106 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:28.109+0000 7f22ca468700 1 -- 172.21.15.58:0/1920844496 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f22c40600e0 msgr2=0x7f22c4106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:28.106 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:28.109+0000 7f22ca468700 1 --2- 172.21.15.58:0/1920844496 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f22c40600e0 0x7f22c4106e90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f22bc004b70 tx=0x7f22bc004d10).stop 2021-11-20T05:37:28.106 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:28.109+0000 7f22ca468700 1 -- 172.21.15.58:0/1920844496 shutdown_connections 2021-11-20T05:37:28.106 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:28.109+0000 7f22ca468700 1 --2- 172.21.15.58:0/1920844496 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f22ac05c2f0 0x7f22ac05e7b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:28.107 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:28.109+0000 7f22ca468700 1 --2- 172.21.15.58:0/1920844496 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f22c40600e0 0x7f22c4106e90 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:28.107 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:28.109+0000 7f22ca468700 1 --2- 172.21.15.58:0/1920844496 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22c40f9720 0x7f22c4107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:28.107 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:28.109+0000 7f22ca468700 1 -- 172.21.15.58:0/1920844496 >> 172.21.15.58:0/1920844496 conn(0x7f22c40f0ba0 msgr2=0x7f22c40f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:28.107 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:28.109+0000 7f22ca468700 1 -- 172.21.15.58:0/1920844496 shutdown_connections 2021-11-20T05:37:28.108 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:28.109+0000 7f22ca468700 1 -- 172.21.15.58:0/1920844496 wait complete. 2021-11-20T05:37:28.108 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 72 2021-11-20T05:37:28.117 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:27.270544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24371, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24371': {'gid': 24371, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 18, 'addr': '172.21.15.187:6827/2249219455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 2249219455}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 2249219455}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:28.118 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24371:up:rejoin 2021-11-20T05:37:28.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:28 smithi058 conmon[33537]: audit 2021-11-20T05:37:28. 2021-11-20T05:37:28.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:28 smithi058 conmon[33537]: 108143+0000 mon.smithi058 (mon.0) 749 : audit [DBG] from='client.? 172.21.15.58:0/1920844496' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:28.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:28 smithi187 conmon[36358]: audit 2021-11-20T05:37:28.108143+0000 mon.smithi058 (mon.0) 749 : audit [DBG] from='client.? 172.21.15.58:0/1920844496' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:28.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:28 smithi187 conmon[36358]: 2021-11-20T05:37:29.119 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:29.251 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.251+0000 7f03fdb68700 1 -- 172.21.15.58:0/432432126 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f03f80f7920 msgr2=0x7f03f80f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:29.251 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.251+0000 7f03fdb68700 1 --2- 172.21.15.58:0/432432126 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f03f80f7920 0x7f03f80f7d80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f03ec004660 tx=0x7f03ec009b30).stop 2021-11-20T05:37:29.251 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.251+0000 7f03fdb68700 1 -- 172.21.15.58:0/432432126 shutdown_connections 2021-11-20T05:37:29.251 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.251+0000 7f03fdb68700 1 --2- 172.21.15.58:0/432432126 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f03f80f7920 0x7f03f80f7d80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:29.252 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.251+0000 7f03fdb68700 1 --2- 172.21.15.58:0/432432126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03f80f6680 0x7f03f80f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:29.252 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.251+0000 7f03fdb68700 1 -- 172.21.15.58:0/432432126 >> 172.21.15.58:0/432432126 conn(0x7f03f80f1de0 msgr2=0x7f03f80f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:29.252 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.251+0000 7f03fdb68700 1 -- 172.21.15.58:0/432432126 shutdown_connections 2021-11-20T05:37:29.252 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.252+0000 7f03fdb68700 1 -- 172.21.15.58:0/432432126 wait complete. 2021-11-20T05:37:29.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.252+0000 7f03fdb68700 1 Processor -- start 2021-11-20T05:37:29.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.252+0000 7f03fdb68700 1 -- start start 2021-11-20T05:37:29.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.252+0000 7f03fdb68700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03f80f6680 0x7f03f8106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:29.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.252+0000 7f03fdb68700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f03f80f7920 0x7f03f8107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:29.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.252+0000 7f03fdb68700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f8107930 con 0x7f03f80f7920 2021-11-20T05:37:29.254 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.252+0000 7f03fdb68700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f8107a70 con 0x7f03f80f6680 2021-11-20T05:37:29.254 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.253+0000 7f03f6ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f03f80f7920 0x7f03f8107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:29.255 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.253+0000 7f03f6ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f03f80f7920 0x7f03f8107380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41680/0 (socket says 172.21.15.58:41680) 2021-11-20T05:37:29.255 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.253+0000 7f03f6ffd700 1 -- 172.21.15.58:0/1457174507 learned_addr learned my addr 172.21.15.58:0/1457174507 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:29.255 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.253+0000 7f03f77fe700 1 --2- 172.21.15.58:0/1457174507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03f80f6680 0x7f03f8106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:29.256 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.253+0000 7f03f6ffd700 1 -- 172.21.15.58:0/1457174507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03f80f6680 msgr2=0x7f03f8106e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:29.256 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.253+0000 7f03f6ffd700 1 --2- 172.21.15.58:0/1457174507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03f80f6680 0x7f03f8106e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:29.256 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.253+0000 7f03f6ffd700 1 -- 172.21.15.58:0/1457174507 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec005040 con 0x7f03f80f7920 2021-11-20T05:37:29.257 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.253+0000 7f03f6ffd700 1 --2- 172.21.15.58:0/1457174507 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f03f80f7920 0x7f03f8107380 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f03ec009f50 tx=0x7f03ec009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:29.257 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.253+0000 7f03dffff700 1 -- 172.21.15.58:0/1457174507 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03ec018070 con 0x7f03f80f7920 2021-11-20T05:37:29.257 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.253+0000 7f03dffff700 1 -- 172.21.15.58:0/1457174507 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03ec013410 con 0x7f03f80f7920 2021-11-20T05:37:29.258 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.253+0000 7f03fdb68700 1 -- 172.21.15.58:0/1457174507 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f810c4b0 con 0x7f03f80f7920 2021-11-20T05:37:29.258 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.253+0000 7f03dffff700 1 -- 172.21.15.58:0/1457174507 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03ec01c3c0 con 0x7f03f80f7920 2021-11-20T05:37:29.258 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.253+0000 7f03fdb68700 1 -- 172.21.15.58:0/1457174507 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f810ca70 con 0x7f03f80f7920 2021-11-20T05:37:29.259 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.254+0000 7f03fdb68700 1 -- 172.21.15.58:0/1457174507 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f8027a00 con 0x7f03f80f7920 2021-11-20T05:37:29.259 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.254+0000 7f03dffff700 1 -- 172.21.15.58:0/1457174507 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f03ec01d420 con 0x7f03f80f7920 2021-11-20T05:37:29.260 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.255+0000 7f03dffff700 1 --2- 172.21.15.58:0/1457174507 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f03e005c2a0 0x7f03e005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:29.261 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.255+0000 7f03dffff700 1 -- 172.21.15.58:0/1457174507 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f03ec077ef0 con 0x7f03f80f7920 2021-11-20T05:37:29.261 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.255+0000 7f03f77fe700 1 --2- 172.21.15.58:0/1457174507 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f03e005c2a0 0x7f03e005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:29.261 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.255+0000 7f03f77fe700 1 --2- 172.21.15.58:0/1457174507 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f03e005c2a0 0x7f03e005e760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f03f80f7780 tx=0x7f03e800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:29.261 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.259+0000 7f03dffff700 1 -- 172.21.15.58:0/1457174507 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03ec04a800 con 0x7f03f80f7920 2021-11-20T05:37:29.455 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.459+0000 7f03fdb68700 1 -- 172.21.15.58:0/1457174507 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03f80fee10 con 0x7f03f80f7920 2021-11-20T05:37:29.456 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.460+0000 7f03dffff700 1 -- 172.21.15.58:0/1457174507 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2576 (secure 0 0 0) 0x7f03ec04de20 con 0x7f03f80f7920 2021-11-20T05:37:29.457 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:29.457 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:27.270544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24371,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24371":{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:29.458 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.461+0000 7f03fdb68700 1 -- 172.21.15.58:0/1457174507 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f03e005c2a0 msgr2=0x7f03e005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:29.459 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.461+0000 7f03fdb68700 1 --2- 172.21.15.58:0/1457174507 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f03e005c2a0 0x7f03e005e760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f03f80f7780 tx=0x7f03e800b040).stop 2021-11-20T05:37:29.459 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.461+0000 7f03fdb68700 1 -- 172.21.15.58:0/1457174507 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f03f80f7920 msgr2=0x7f03f8107380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:29.459 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.461+0000 7f03fdb68700 1 --2- 172.21.15.58:0/1457174507 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f03f80f7920 0x7f03f8107380 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f03ec009f50 tx=0x7f03ec009b30).stop 2021-11-20T05:37:29.459 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.461+0000 7f03fdb68700 1 -- 172.21.15.58:0/1457174507 shutdown_connections 2021-11-20T05:37:29.460 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.461+0000 7f03fdb68700 1 --2- 172.21.15.58:0/1457174507 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f03e005c2a0 0x7f03e005e760 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:29.460 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.461+0000 7f03fdb68700 1 --2- 172.21.15.58:0/1457174507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03f80f6680 0x7f03f8106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:29.460 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.461+0000 7f03fdb68700 1 --2- 172.21.15.58:0/1457174507 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f03f80f7920 0x7f03f8107380 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:29.460 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.461+0000 7f03fdb68700 1 -- 172.21.15.58:0/1457174507 >> 172.21.15.58:0/1457174507 conn(0x7f03f80f1de0 msgr2=0x7f03f80fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:29.460 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.461+0000 7f03fdb68700 1 -- 172.21.15.58:0/1457174507 shutdown_connections 2021-11-20T05:37:29.461 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:29.462+0000 7f03fdb68700 1 -- 172.21.15.58:0/1457174507 wait complete. 2021-11-20T05:37:29.461 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 72 2021-11-20T05:37:29.470 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:27.270544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24371, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24371': {'gid': 24371, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 18, 'addr': '172.21.15.187:6827/2249219455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 2249219455}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 2249219455}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:29.471 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24371:up:rejoin 2021-11-20T05:37:29.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:29 smithi187 conmon[36358]: cluster 2021-11-20T05:37:28.284774+0000 mgr.smithi058.zagdlk (mgr.14164) 293 : cluster [DBG] pgmap v266: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2021-11-20T05:37:29.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:29 smithi187 conmon[36358]: audit 2021-11-20T05:37:29.318761+0000 mon.smithi058 (mon.0) 750 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:37:29.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:29 smithi058 conmon[33537]: cluster 2021-11-20T05:37:28.284774+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:37:29.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:29 smithi058 conmon[33537]: 293 : cluster [DBG] pgmap v266: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2021-11-20T05:37:29.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:29 smithi058 conmon[33537]: audit 2021-11-20T05:37:29.318761+0000 mon.smithi058 (mon.0) 750 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:37:30.472 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:30.501 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:30 smithi058 conmon[33537]: audit 2021-11-20T05:37:29.460642+0000 mon.smithi058 (mon 2021-11-20T05:37:30.502 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:30 smithi058 conmon[33537]: .0) 751 : audit [DBG] from='client.? 172.21.15.58:0/1457174507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:30.502 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:30 smithi058 conmon[33537]: audit 2021-11-20T05:37:29.701551+0000 mon.smithi058 (mon.0) 752 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:37:30.608 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.610+0000 7f85c2916700 1 -- 172.21.15.58:0/3089233440 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f85bc0f6820 msgr2=0x7f85bc0f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:30.609 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.610+0000 7f85c2916700 1 --2- 172.21.15.58:0/3089233440 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f85bc0f6820 0x7f85bc0f6c80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f85b0004660 tx=0x7f85b0009b30).stop 2021-11-20T05:37:30.609 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.610+0000 7f85c2916700 1 -- 172.21.15.58:0/3089233440 shutdown_connections 2021-11-20T05:37:30.609 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.610+0000 7f85c2916700 1 --2- 172.21.15.58:0/3089233440 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f85bc0f6820 0x7f85bc0f6c80 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:30.610 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.610+0000 7f85c2916700 1 --2- 172.21.15.58:0/3089233440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85bc0f5580 0x7f85bc0f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:30.610 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.610+0000 7f85c2916700 1 -- 172.21.15.58:0/3089233440 >> 172.21.15.58:0/3089233440 conn(0x7f85bc0f0ce0 msgr2=0x7f85bc0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:30.610 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.610+0000 7f85c2916700 1 -- 172.21.15.58:0/3089233440 shutdown_connections 2021-11-20T05:37:30.610 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.610+0000 7f85c2916700 1 -- 172.21.15.58:0/3089233440 wait complete. 2021-11-20T05:37:30.611 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.610+0000 7f85c2916700 1 Processor -- start 2021-11-20T05:37:30.611 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.611+0000 7f85c2916700 1 -- start start 2021-11-20T05:37:30.611 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.611+0000 7f85c2916700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85bc0f5580 0x7f85bc060510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:30.611 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.611+0000 7f85c2916700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f85bc0f6820 0x7f85bc10bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:30.611 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.611+0000 7f85c2916700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc10c100 con 0x7f85bc0f6820 2021-11-20T05:37:30.612 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.611+0000 7f85c2916700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc10c240 con 0x7f85bc0f5580 2021-11-20T05:37:30.612 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.611+0000 7f85bb7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f85bc0f6820 0x7f85bc10bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:30.612 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.611+0000 7f85bb7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f85bc0f6820 0x7f85bc10bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41694/0 (socket says 172.21.15.58:41694) 2021-11-20T05:37:30.612 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.611+0000 7f85bb7fe700 1 -- 172.21.15.58:0/3076895287 learned_addr learned my addr 172.21.15.58:0/3076895287 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:30.613 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.611+0000 7f85bbfff700 1 --2- 172.21.15.58:0/3076895287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85bc0f5580 0x7f85bc060510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:30.613 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.611+0000 7f85bb7fe700 1 -- 172.21.15.58:0/3076895287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85bc0f5580 msgr2=0x7f85bc060510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:30.613 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.611+0000 7f85bb7fe700 1 --2- 172.21.15.58:0/3076895287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85bc0f5580 0x7f85bc060510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:30.613 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.611+0000 7f85bb7fe700 1 -- 172.21.15.58:0/3076895287 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b0005040 con 0x7f85bc0f6820 2021-11-20T05:37:30.614 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.611+0000 7f85bb7fe700 1 --2- 172.21.15.58:0/3076895287 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f85bc0f6820 0x7f85bc10bc10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f85b0009870 tx=0x7f85b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:30.614 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.611+0000 7f85b8ff9700 1 -- 172.21.15.58:0/3076895287 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85b0019070 con 0x7f85bc0f6820 2021-11-20T05:37:30.614 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.611+0000 7f85b8ff9700 1 -- 172.21.15.58:0/3076895287 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f85b0007e70 con 0x7f85bc0f6820 2021-11-20T05:37:30.614 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.612+0000 7f85c2916700 1 -- 172.21.15.58:0/3076895287 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85bc10c440 con 0x7f85bc0f6820 2021-11-20T05:37:30.615 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.612+0000 7f85b8ff9700 1 -- 172.21.15.58:0/3076895287 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85b0014c90 con 0x7f85bc0f6820 2021-11-20T05:37:30.615 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.612+0000 7f85c2916700 1 -- 172.21.15.58:0/3076895287 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85bc10c9a0 con 0x7f85bc0f6820 2021-11-20T05:37:30.616 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.613+0000 7f85c2916700 1 -- 172.21.15.58:0/3076895287 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85bc027a00 con 0x7f85bc0f6820 2021-11-20T05:37:30.617 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.613+0000 7f85b8ff9700 1 -- 172.21.15.58:0/3076895287 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f85b0025070 con 0x7f85bc0f6820 2021-11-20T05:37:30.617 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.613+0000 7f85b8ff9700 1 --2- 172.21.15.58:0/3076895287 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f85a405c2a0 0x7f85a405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:30.617 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.613+0000 7f85b8ff9700 1 -- 172.21.15.58:0/3076895287 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f85b00772d0 con 0x7f85bc0f6820 2021-11-20T05:37:30.617 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.613+0000 7f85bbfff700 1 --2- 172.21.15.58:0/3076895287 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f85a405c2a0 0x7f85a405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:30.617 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.614+0000 7f85bbfff700 1 --2- 172.21.15.58:0/3076895287 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f85a405c2a0 0x7f85a405e760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f85ac009fd0 tx=0x7f85ac00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:30.618 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.617+0000 7f85b8ff9700 1 -- 172.21.15.58:0/3076895287 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85b0049be0 con 0x7f85bc0f6820 2021-11-20T05:37:30.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:30 smithi187 conmon[36358]: audit 2021-11-20T05:37:29.460642+0000 mon.smithi058 (mon.0) 751 : audit [DBG] from='client.? 172.21.15.58:0/1457174507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:30.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:30 smithi187 conmon[36358]: audit 2021-11-20T05:37:29.701551+0000 mon.smithi058 (mon.0) 752 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:37:30.811 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.815+0000 7f85c2916700 1 -- 172.21.15.58:0/3076895287 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85bc0fdd10 con 0x7f85bc0f6820 2021-11-20T05:37:30.812 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.815+0000 7f85b8ff9700 1 -- 172.21.15.58:0/3076895287 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2576 (secure 0 0 0) 0x7f85b004d200 con 0x7f85bc0f6820 2021-11-20T05:37:30.812 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:30.813 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:27.270544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24371,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24371":{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:30.814 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.817+0000 7f85c2916700 1 -- 172.21.15.58:0/3076895287 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f85a405c2a0 msgr2=0x7f85a405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:30.814 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.817+0000 7f85c2916700 1 --2- 172.21.15.58:0/3076895287 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f85a405c2a0 0x7f85a405e760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f85ac009fd0 tx=0x7f85ac00b040).stop 2021-11-20T05:37:30.815 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.817+0000 7f85c2916700 1 -- 172.21.15.58:0/3076895287 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f85bc0f6820 msgr2=0x7f85bc10bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:30.815 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.817+0000 7f85c2916700 1 --2- 172.21.15.58:0/3076895287 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f85bc0f6820 0x7f85bc10bc10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f85b0009870 tx=0x7f85b000b040).stop 2021-11-20T05:37:30.815 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.817+0000 7f85c2916700 1 -- 172.21.15.58:0/3076895287 shutdown_connections 2021-11-20T05:37:30.815 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.817+0000 7f85c2916700 1 --2- 172.21.15.58:0/3076895287 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f85a405c2a0 0x7f85a405e760 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:30.816 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.817+0000 7f85c2916700 1 --2- 172.21.15.58:0/3076895287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f85bc0f5580 0x7f85bc060510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:30.816 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.817+0000 7f85c2916700 1 --2- 172.21.15.58:0/3076895287 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f85bc0f6820 0x7f85bc10bc10 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:30.816 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.817+0000 7f85c2916700 1 -- 172.21.15.58:0/3076895287 >> 172.21.15.58:0/3076895287 conn(0x7f85bc0f0ce0 msgr2=0x7f85bc0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:30.816 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.817+0000 7f85c2916700 1 -- 172.21.15.58:0/3076895287 shutdown_connections 2021-11-20T05:37:30.817 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:30.817+0000 7f85c2916700 1 -- 172.21.15.58:0/3076895287 wait complete. 2021-11-20T05:37:30.817 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 72 2021-11-20T05:37:30.826 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:27.270544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24371, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24371': {'gid': 24371, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 18, 'addr': '172.21.15.187:6827/2249219455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 2249219455}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 2249219455}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:30.827 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24371:up:rejoin 2021-11-20T05:37:31.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:31 smithi187 conmon[36358]: cluster 2021-11-20T05:37:30.285398+0000 mgr.smithi058.zagdlk (mgr.14164) 294 : cluster [DBG] pgmap v267: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:31.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:31 smithi187 conmon[36358]: audit 2021-11-20T05:37:30.816372+0000 mon.smithi058 (mon.0) 753 : audit [DBG] from='client.? 172.21.15.58:0/3076895287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:31.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:31 smithi058 conmon[33537]: cluster 2021-11-20T05:37:30.285398+0000 mgr.smithi058.zagdlk (mgr.14164) 294 : cluster [DBG] pgmap v267: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:31.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:31 smithi058 conmon[33537]: audit 2021-11-20T05:37:30.816372+0000 mon.smithi058 (mon.0) 753 : audit [DBG] from='client.? 172.21.15.58:0/3076895287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:31.828 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:31.969 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.964+0000 7f9a19d16700 1 -- 172.21.15.58:0/1756977128 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9a14060140 msgr2=0x7f9a140f9280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:31.969 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.964+0000 7f9a19d16700 1 --2- 172.21.15.58:0/1756977128 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9a14060140 0x7f9a140f9280 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f9a04004660 tx=0x7f9a04009b30).stop 2021-11-20T05:37:31.970 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.964+0000 7f9a19d16700 1 -- 172.21.15.58:0/1756977128 shutdown_connections 2021-11-20T05:37:31.970 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.964+0000 7f9a19d16700 1 --2- 172.21.15.58:0/1756977128 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a140f9800 0x7f9a140fbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:31.970 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.964+0000 7f9a19d16700 1 --2- 172.21.15.58:0/1756977128 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9a14060140 0x7f9a140f9280 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:31.971 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.964+0000 7f9a19d16700 1 -- 172.21.15.58:0/1756977128 >> 172.21.15.58:0/1756977128 conn(0x7f9a140f0ce0 msgr2=0x7f9a140f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:31.971 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.964+0000 7f9a19d16700 1 -- 172.21.15.58:0/1756977128 shutdown_connections 2021-11-20T05:37:31.971 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.964+0000 7f9a19d16700 1 -- 172.21.15.58:0/1756977128 wait complete. 2021-11-20T05:37:31.971 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.965+0000 7f9a19d16700 1 Processor -- start 2021-11-20T05:37:31.972 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.965+0000 7f9a19d16700 1 -- start start 2021-11-20T05:37:31.972 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.965+0000 7f9a19d16700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9a14060140 0x7f9a14106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:31.972 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.965+0000 7f9a19d16700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a140f9800 0x7f9a14107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:31.972 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.965+0000 7f9a19d16700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a14107930 con 0x7f9a14060140 2021-11-20T05:37:31.973 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.965+0000 7f9a19d16700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a14107a70 con 0x7f9a140f9800 2021-11-20T05:37:31.974 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.965+0000 7f9a137fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9a14060140 0x7f9a14106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:31.974 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.965+0000 7f9a137fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9a14060140 0x7f9a14106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41702/0 (socket says 172.21.15.58:41702) 2021-11-20T05:37:31.975 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.965+0000 7f9a137fe700 1 -- 172.21.15.58:0/2043452183 learned_addr learned my addr 172.21.15.58:0/2043452183 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:31.976 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.966+0000 7f9a12ffd700 1 --2- 172.21.15.58:0/2043452183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a140f9800 0x7f9a14107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:31.976 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.966+0000 7f9a137fe700 1 -- 172.21.15.58:0/2043452183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a140f9800 msgr2=0x7f9a14107380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:31.976 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.966+0000 7f9a137fe700 1 --2- 172.21.15.58:0/2043452183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a140f9800 0x7f9a14107380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:31.976 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.966+0000 7f9a137fe700 1 -- 172.21.15.58:0/2043452183 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a04005040 con 0x7f9a14060140 2021-11-20T05:37:31.977 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.966+0000 7f9a137fe700 1 --2- 172.21.15.58:0/2043452183 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9a14060140 0x7f9a14106e90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f9a04009830 tx=0x7f9a04009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:31.977 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.966+0000 7f99fbfff700 1 -- 172.21.15.58:0/2043452183 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a04018070 con 0x7f9a14060140 2021-11-20T05:37:31.977 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.966+0000 7f99fbfff700 1 -- 172.21.15.58:0/2043452183 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a04013410 con 0x7f9a14060140 2021-11-20T05:37:31.977 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.966+0000 7f9a19d16700 1 -- 172.21.15.58:0/2043452183 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1410c4b0 con 0x7f9a14060140 2021-11-20T05:37:31.978 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.966+0000 7f99fbfff700 1 -- 172.21.15.58:0/2043452183 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a0401c400 con 0x7f9a14060140 2021-11-20T05:37:31.978 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.966+0000 7f9a19d16700 1 -- 172.21.15.58:0/2043452183 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a1410ca70 con 0x7f9a14060140 2021-11-20T05:37:31.978 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.967+0000 7f9a19d16700 1 -- 172.21.15.58:0/2043452183 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a14027a00 con 0x7f9a14060140 2021-11-20T05:37:31.978 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.967+0000 7f99fbfff700 1 -- 172.21.15.58:0/2043452183 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9a0401d420 con 0x7f9a14060140 2021-11-20T05:37:31.978 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.968+0000 7f99fbfff700 1 --2- 172.21.15.58:0/2043452183 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f99fc05c2a0 0x7f99fc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:31.979 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.968+0000 7f99fbfff700 1 -- 172.21.15.58:0/2043452183 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9a040780f0 con 0x7f9a14060140 2021-11-20T05:37:31.979 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.968+0000 7f9a12ffd700 1 --2- 172.21.15.58:0/2043452183 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f99fc05c2a0 0x7f99fc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:31.979 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.968+0000 7f9a12ffd700 1 --2- 172.21.15.58:0/2043452183 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f99fc05c2a0 0x7f99fc05e760 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9a08007fd0 tx=0x7f9a08009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:31.979 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:31.972+0000 7f99fbfff700 1 -- 172.21.15.58:0/2043452183 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a0404aa00 con 0x7f9a14060140 2021-11-20T05:37:32.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:32.169+0000 7f9a19d16700 1 -- 172.21.15.58:0/2043452183 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a140fee50 con 0x7f9a14060140 2021-11-20T05:37:32.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:32.170+0000 7f99fbfff700 1 -- 172.21.15.58:0/2043452183 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2576 (secure 0 0 0) 0x7f9a0404e020 con 0x7f9a14060140 2021-11-20T05:37:32.167 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:32.167 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:27.270544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24371,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24371":{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:32.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:32.171+0000 7f9a19d16700 1 -- 172.21.15.58:0/2043452183 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f99fc05c2a0 msgr2=0x7f99fc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:32.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:32.171+0000 7f9a19d16700 1 --2- 172.21.15.58:0/2043452183 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f99fc05c2a0 0x7f99fc05e760 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9a08007fd0 tx=0x7f9a08009040).stop 2021-11-20T05:37:32.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:32.171+0000 7f9a19d16700 1 -- 172.21.15.58:0/2043452183 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9a14060140 msgr2=0x7f9a14106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:32.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:32.171+0000 7f9a19d16700 1 --2- 172.21.15.58:0/2043452183 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9a14060140 0x7f9a14106e90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f9a04009830 tx=0x7f9a04009b30).stop 2021-11-20T05:37:32.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:32.172+0000 7f9a19d16700 1 -- 172.21.15.58:0/2043452183 shutdown_connections 2021-11-20T05:37:32.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:32.172+0000 7f9a19d16700 1 --2- 172.21.15.58:0/2043452183 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f99fc05c2a0 0x7f99fc05e760 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:32.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:32.172+0000 7f9a19d16700 1 --2- 172.21.15.58:0/2043452183 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9a14060140 0x7f9a14106e90 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:32.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:32.172+0000 7f9a19d16700 1 --2- 172.21.15.58:0/2043452183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a140f9800 0x7f9a14107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:32.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:32.172+0000 7f9a19d16700 1 -- 172.21.15.58:0/2043452183 >> 172.21.15.58:0/2043452183 conn(0x7f9a140f0ce0 msgr2=0x7f9a140f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:32.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:32.172+0000 7f9a19d16700 1 -- 172.21.15.58:0/2043452183 shutdown_connections 2021-11-20T05:37:32.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:32.172+0000 7f9a19d16700 1 -- 172.21.15.58:0/2043452183 wait complete. 2021-11-20T05:37:32.171 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 72 2021-11-20T05:37:32.180 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:27.270544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24371, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24371': {'gid': 24371, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 18, 'addr': '172.21.15.187:6827/2249219455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 2249219455}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 2249219455}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:32.181 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24371:up:rejoin 2021-11-20T05:37:32.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:32 smithi058 conmon[33537]: audit 2021-11-20T05:37:32.170995+0000 2021-11-20T05:37:32.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:32 smithi058 conmon[33537]: mon.smithi058 (mon.0) 754 : audit [DBG] from='client.? 172.21.15.58:0/2043452183' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:32.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:32 smithi187 conmon[36358]: audit 2021-11-20T05:37:32.170995+0000 mon.smithi058 (mon.0) 754 : audit [DBG] from='client.? 172.21.15.58:0/2043452183' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:33.181 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:33.316 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.317+0000 7feb85d52700 1 -- 172.21.15.58:0/4279230697 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7feb800f7920 msgr2=0x7feb800f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:33.316 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.317+0000 7feb85d52700 1 --2- 172.21.15.58:0/4279230697 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7feb800f7920 0x7feb800f7d80 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7feb68004660 tx=0x7feb68009b30).stop 2021-11-20T05:37:33.317 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.317+0000 7feb85d52700 1 -- 172.21.15.58:0/4279230697 shutdown_connections 2021-11-20T05:37:33.317 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.317+0000 7feb85d52700 1 --2- 172.21.15.58:0/4279230697 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7feb800f7920 0x7feb800f7d80 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:33.317 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.317+0000 7feb85d52700 1 --2- 172.21.15.58:0/4279230697 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb800f6680 0x7feb800f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:33.317 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.317+0000 7feb85d52700 1 -- 172.21.15.58:0/4279230697 >> 172.21.15.58:0/4279230697 conn(0x7feb800f1de0 msgr2=0x7feb800f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:33.318 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.317+0000 7feb85d52700 1 -- 172.21.15.58:0/4279230697 shutdown_connections 2021-11-20T05:37:33.318 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.318+0000 7feb85d52700 1 -- 172.21.15.58:0/4279230697 wait complete. 2021-11-20T05:37:33.318 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.318+0000 7feb85d52700 1 Processor -- start 2021-11-20T05:37:33.318 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.318+0000 7feb85d52700 1 -- start start 2021-11-20T05:37:33.318 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.318+0000 7feb85d52700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb800f6680 0x7feb8006b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:33.319 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.318+0000 7feb85d52700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7feb800f7920 0x7feb80069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:33.319 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.318+0000 7feb85d52700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8006bd70 con 0x7feb800f7920 2021-11-20T05:37:33.319 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.318+0000 7feb85d52700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8006a310 con 0x7feb800f6680 2021-11-20T05:37:33.319 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.319+0000 7feb7effd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7feb800f7920 0x7feb80069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:33.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.319+0000 7feb7effd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7feb800f7920 0x7feb80069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41716/0 (socket says 172.21.15.58:41716) 2021-11-20T05:37:33.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.319+0000 7feb7effd700 1 -- 172.21.15.58:0/996167100 learned_addr learned my addr 172.21.15.58:0/996167100 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:33.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.319+0000 7feb7f7fe700 1 --2- 172.21.15.58:0/996167100 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb800f6680 0x7feb8006b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:33.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.319+0000 7feb7effd700 1 -- 172.21.15.58:0/996167100 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb800f6680 msgr2=0x7feb8006b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:33.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.319+0000 7feb7effd700 1 --2- 172.21.15.58:0/996167100 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb800f6680 0x7feb8006b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:33.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.319+0000 7feb7effd700 1 -- 172.21.15.58:0/996167100 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb68005040 con 0x7feb800f7920 2021-11-20T05:37:33.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.319+0000 7feb7f7fe700 1 --2- 172.21.15.58:0/996167100 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb800f6680 0x7feb8006b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:37:33.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.319+0000 7feb7effd700 1 --2- 172.21.15.58:0/996167100 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7feb800f7920 0x7feb80069e20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7feb68000c00 tx=0x7feb6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:33.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.319+0000 7feb77fff700 1 -- 172.21.15.58:0/996167100 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb68019070 con 0x7feb800f7920 2021-11-20T05:37:33.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.319+0000 7feb85d52700 1 -- 172.21.15.58:0/996167100 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb8006a530 con 0x7feb800f7920 2021-11-20T05:37:33.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.319+0000 7feb77fff700 1 -- 172.21.15.58:0/996167100 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb68007e70 con 0x7feb800f7920 2021-11-20T05:37:33.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.319+0000 7feb77fff700 1 -- 172.21.15.58:0/996167100 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb68014d00 con 0x7feb800f7920 2021-11-20T05:37:33.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.319+0000 7feb85d52700 1 -- 172.21.15.58:0/996167100 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8006aa90 con 0x7feb800f7920 2021-11-20T05:37:33.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.320+0000 7feb85d52700 1 -- 172.21.15.58:0/996167100 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb80046330 con 0x7feb800f7920 2021-11-20T05:37:33.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.321+0000 7feb77fff700 1 -- 172.21.15.58:0/996167100 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7feb68025070 con 0x7feb800f7920 2021-11-20T05:37:33.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.321+0000 7feb77fff700 1 --2- 172.21.15.58:0/996167100 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7feb6c05c2a0 0x7feb6c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:33.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.321+0000 7feb77fff700 1 -- 172.21.15.58:0/996167100 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7feb680772e0 con 0x7feb800f7920 2021-11-20T05:37:33.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.321+0000 7feb7f7fe700 1 --2- 172.21.15.58:0/996167100 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7feb6c05c2a0 0x7feb6c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:33.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.321+0000 7feb7f7fe700 1 --2- 172.21.15.58:0/996167100 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7feb6c05c2a0 0x7feb6c05e760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7feb70004fd0 tx=0x7feb7000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:33.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.325+0000 7feb77fff700 1 -- 172.21.15.58:0/996167100 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb68049b90 con 0x7feb800f7920 2021-11-20T05:37:33.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:33 smithi058 conmon[33537]: cluster 2021-11-20T05:37:32.285936+0000 2021-11-20T05:37:33.471 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:33 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 295 : cluster [DBG] pgmap v268: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:33.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.519+0000 7feb85d52700 1 -- 172.21.15.58:0/996167100 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb80027a00 con 0x7feb800f7920 2021-11-20T05:37:33.516 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.519+0000 7feb77fff700 1 -- 172.21.15.58:0/996167100 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2576 (secure 0 0 0) 0x7feb68023070 con 0x7feb800f7920 2021-11-20T05:37:33.516 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:33.517 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:27.270544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24371,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24371":{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:33.518 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.521+0000 7feb85d52700 1 -- 172.21.15.58:0/996167100 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7feb6c05c2a0 msgr2=0x7feb6c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:33.518 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.521+0000 7feb85d52700 1 --2- 172.21.15.58:0/996167100 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7feb6c05c2a0 0x7feb6c05e760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7feb70004fd0 tx=0x7feb7000b040).stop 2021-11-20T05:37:33.518 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.521+0000 7feb85d52700 1 -- 172.21.15.58:0/996167100 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7feb800f7920 msgr2=0x7feb80069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:33.519 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.521+0000 7feb85d52700 1 --2- 172.21.15.58:0/996167100 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7feb800f7920 0x7feb80069e20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7feb68000c00 tx=0x7feb6800b040).stop 2021-11-20T05:37:33.519 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.521+0000 7feb85d52700 1 -- 172.21.15.58:0/996167100 shutdown_connections 2021-11-20T05:37:33.519 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.521+0000 7feb85d52700 1 --2- 172.21.15.58:0/996167100 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7feb6c05c2a0 0x7feb6c05e760 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:33.519 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.521+0000 7feb85d52700 1 --2- 172.21.15.58:0/996167100 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feb800f6680 0x7feb8006b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:33.520 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.521+0000 7feb85d52700 1 --2- 172.21.15.58:0/996167100 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7feb800f7920 0x7feb80069e20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:33.520 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.521+0000 7feb85d52700 1 -- 172.21.15.58:0/996167100 >> 172.21.15.58:0/996167100 conn(0x7feb800f1de0 msgr2=0x7feb800fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:33.520 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.521+0000 7feb85d52700 1 -- 172.21.15.58:0/996167100 shutdown_connections 2021-11-20T05:37:33.520 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:33.521+0000 7feb85d52700 1 -- 172.21.15.58:0/996167100 wait complete. 2021-11-20T05:37:33.521 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 72 2021-11-20T05:37:33.530 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:27.270544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24371, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24371': {'gid': 24371, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 18, 'addr': '172.21.15.187:6827/2249219455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 2249219455}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 2249219455}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:33.530 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24371:up:rejoin 2021-11-20T05:37:33.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:33 smithi187 conmon[36358]: cluster 2021-11-20T05:37:32.285936+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:37:33.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:33 smithi187 conmon[36358]: ) 295 : cluster [DBG] pgmap v268: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:34.531 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:34.665 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.667+0000 7fb1391af700 1 -- 172.21.15.58:0/3831691481 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb1340f6820 msgr2=0x7fb1340f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:34.666 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.667+0000 7fb1391af700 1 --2- 172.21.15.58:0/3831691481 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb1340f6820 0x7fb1340f6c80 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fb124004660 tx=0x7fb124009b30).stop 2021-11-20T05:37:34.666 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.667+0000 7fb1391af700 1 -- 172.21.15.58:0/3831691481 shutdown_connections 2021-11-20T05:37:34.666 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.667+0000 7fb1391af700 1 --2- 172.21.15.58:0/3831691481 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb1340f6820 0x7fb1340f6c80 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:34.667 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.667+0000 7fb1391af700 1 --2- 172.21.15.58:0/3831691481 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1340f5580 0x7fb1340f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:34.667 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.667+0000 7fb1391af700 1 -- 172.21.15.58:0/3831691481 >> 172.21.15.58:0/3831691481 conn(0x7fb1340f0ce0 msgr2=0x7fb1340f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:34.667 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.667+0000 7fb1391af700 1 -- 172.21.15.58:0/3831691481 shutdown_connections 2021-11-20T05:37:34.668 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.667+0000 7fb1391af700 1 -- 172.21.15.58:0/3831691481 wait complete. 2021-11-20T05:37:34.668 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.668+0000 7fb1391af700 1 Processor -- start 2021-11-20T05:37:34.668 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.668+0000 7fb1391af700 1 -- start start 2021-11-20T05:37:34.668 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.668+0000 7fb1391af700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1340f5580 0x7fb134105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:34.669 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.668+0000 7fb1391af700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb1340f6820 0x7fb134106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:34.669 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.668+0000 7fb1391af700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb134106890 con 0x7fb1340f6820 2021-11-20T05:37:34.669 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.668+0000 7fb1391af700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1341069d0 con 0x7fb1340f5580 2021-11-20T05:37:34.669 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.668+0000 7fb13259c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb1340f6820 0x7fb134106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:34.670 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.668+0000 7fb13259c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb1340f6820 0x7fb134106290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41728/0 (socket says 172.21.15.58:41728) 2021-11-20T05:37:34.670 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.668+0000 7fb13259c700 1 -- 172.21.15.58:0/17451448 learned_addr learned my addr 172.21.15.58:0/17451448 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:34.670 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.668+0000 7fb132d9d700 1 --2- 172.21.15.58:0/17451448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1340f5580 0x7fb134105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:34.670 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.668+0000 7fb13259c700 1 -- 172.21.15.58:0/17451448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1340f5580 msgr2=0x7fb134105da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:34.671 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.668+0000 7fb13259c700 1 --2- 172.21.15.58:0/17451448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1340f5580 0x7fb134105da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:34.671 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.668+0000 7fb13259c700 1 -- 172.21.15.58:0/17451448 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb124005040 con 0x7fb1340f6820 2021-11-20T05:37:34.671 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.668+0000 7fb13259c700 1 --2- 172.21.15.58:0/17451448 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb1340f6820 0x7fb134106290 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fb124009f50 tx=0x7fb124009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:34.671 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.669+0000 7fb11b7fe700 1 -- 172.21.15.58:0/17451448 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb124018070 con 0x7fb1340f6820 2021-11-20T05:37:34.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.669+0000 7fb1391af700 1 -- 172.21.15.58:0/17451448 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13410b380 con 0x7fb1340f6820 2021-11-20T05:37:34.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.669+0000 7fb11b7fe700 1 -- 172.21.15.58:0/17451448 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb124013410 con 0x7fb1340f6820 2021-11-20T05:37:34.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.669+0000 7fb11b7fe700 1 -- 172.21.15.58:0/17451448 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb12401c3c0 con 0x7fb1340f6820 2021-11-20T05:37:34.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.669+0000 7fb1391af700 1 -- 172.21.15.58:0/17451448 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13410b8e0 con 0x7fb1340f6820 2021-11-20T05:37:34.674 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.670+0000 7fb1391af700 1 -- 172.21.15.58:0/17451448 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb134027a00 con 0x7fb1340f6820 2021-11-20T05:37:34.674 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.670+0000 7fb11b7fe700 1 -- 172.21.15.58:0/17451448 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb12401d420 con 0x7fb1340f6820 2021-11-20T05:37:34.674 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.670+0000 7fb11b7fe700 1 --2- 172.21.15.58:0/17451448 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb11c05c2a0 0x7fb11c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:34.674 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.670+0000 7fb11b7fe700 1 -- 172.21.15.58:0/17451448 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb124077f70 con 0x7fb1340f6820 2021-11-20T05:37:34.675 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.670+0000 7fb132d9d700 1 --2- 172.21.15.58:0/17451448 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb11c05c2a0 0x7fb11c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:34.675 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.671+0000 7fb132d9d700 1 --2- 172.21.15.58:0/17451448 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb11c05c2a0 0x7fb11c05e760 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fb134107290 tx=0x7fb12800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:34.675 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.674+0000 7fb11b7fe700 1 -- 172.21.15.58:0/17451448 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1240a42b0 con 0x7fb1340f6820 2021-11-20T05:37:34.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:34 smithi187 conmon[36358]: audit 2021-11-20T05:37:33.520559+0000 mon.smithi058 (mon.0) 755 : audit [DBG] 2021-11-20T05:37:34.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:34 smithi187 conmon[36358]: from='client.? 172.21.15.58:0/996167100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:34.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:34 smithi058 conmon[33537]: audit 2021-11-20T05:37:33.520559+0000 mon.smithi058 ( 2021-11-20T05:37:34.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:34 smithi058 conmon[33537]: mon.0) 755 : audit [DBG] from='client.? 172.21.15.58:0/996167100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:34.869 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.872+0000 7fb1391af700 1 -- 172.21.15.58:0/17451448 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1340fdd10 con 0x7fb1340f6820 2021-11-20T05:37:34.869 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.873+0000 7fb11b7fe700 1 -- 172.21.15.58:0/17451448 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2576 (secure 0 0 0) 0x7fb12404a880 con 0x7fb1340f6820 2021-11-20T05:37:34.870 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:34.870 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:27.270544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24371,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24371":{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:34.871 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.874+0000 7fb1391af700 1 -- 172.21.15.58:0/17451448 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb11c05c2a0 msgr2=0x7fb11c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:34.872 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.874+0000 7fb1391af700 1 --2- 172.21.15.58:0/17451448 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb11c05c2a0 0x7fb11c05e760 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fb134107290 tx=0x7fb12800b040).stop 2021-11-20T05:37:34.872 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.874+0000 7fb1391af700 1 -- 172.21.15.58:0/17451448 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb1340f6820 msgr2=0x7fb134106290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:34.872 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.874+0000 7fb1391af700 1 --2- 172.21.15.58:0/17451448 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb1340f6820 0x7fb134106290 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fb124009f50 tx=0x7fb124009b30).stop 2021-11-20T05:37:34.872 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.874+0000 7fb1391af700 1 -- 172.21.15.58:0/17451448 shutdown_connections 2021-11-20T05:37:34.873 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.874+0000 7fb1391af700 1 --2- 172.21.15.58:0/17451448 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb11c05c2a0 0x7fb11c05e760 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:34.873 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.874+0000 7fb1391af700 1 --2- 172.21.15.58:0/17451448 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb1340f5580 0x7fb134105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:34.873 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.874+0000 7fb1391af700 1 --2- 172.21.15.58:0/17451448 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb1340f6820 0x7fb134106290 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:34.873 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.874+0000 7fb1391af700 1 -- 172.21.15.58:0/17451448 >> 172.21.15.58:0/17451448 conn(0x7fb1340f0ce0 msgr2=0x7fb1340fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:34.873 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.874+0000 7fb1391af700 1 -- 172.21.15.58:0/17451448 shutdown_connections 2021-11-20T05:37:34.874 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:34.874+0000 7fb1391af700 1 -- 172.21.15.58:0/17451448 wait complete. 2021-11-20T05:37:34.874 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 72 2021-11-20T05:37:34.883 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:27.270544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24371, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24371': {'gid': 24371, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 18, 'addr': '172.21.15.187:6827/2249219455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 2249219455}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 2249219455}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:34.883 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24371:up:rejoin 2021-11-20T05:37:35.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:35 smithi187 conmon[36358]: cluster 2021-11-20T05:37:34.286394+0000 mgr.smithi058.zagdlk (mgr.14164) 296 : cluster 2021-11-20T05:37:35.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:35 smithi187 conmon[36358]: [DBG] pgmap v269: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:35.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:35 smithi187 conmon[36358]: audit 2021-11-20T05:37:34.873863+0000 mon.smithi058 (mon.0) 756 : audit [DBG] from='client.? 172.21.15.58:0/17451448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:35.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:35 smithi058 conmon[33537]: cluster 2021-11-20T05:37:34.286394+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:37:35.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:35 smithi058 conmon[33537]: 296 : cluster [DBG] pgmap v269: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:35.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:35 smithi058 conmon[33537]: audit 2021-11-20T05:37:34.873863+0000 mon.smithi058 (mon.0) 756 : audit [DBG] from='client.? 172.21.15.58:0/17451448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:35.884 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:36.019 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.020+0000 7efbfb7a7700 1 -- 172.21.15.58:0/1816171822 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbf40f6820 msgr2=0x7efbf40f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:36.019 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.020+0000 7efbfb7a7700 1 --2- 172.21.15.58:0/1816171822 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbf40f6820 0x7efbf40f6c80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7efbf0004660 tx=0x7efbf0009b30).stop 2021-11-20T05:37:36.020 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.020+0000 7efbfb7a7700 1 -- 172.21.15.58:0/1816171822 shutdown_connections 2021-11-20T05:37:36.020 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.020+0000 7efbfb7a7700 1 --2- 172.21.15.58:0/1816171822 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbf40f6820 0x7efbf40f6c80 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:36.020 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.020+0000 7efbfb7a7700 1 --2- 172.21.15.58:0/1816171822 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbf40f5580 0x7efbf40f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:36.020 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.020+0000 7efbfb7a7700 1 -- 172.21.15.58:0/1816171822 >> 172.21.15.58:0/1816171822 conn(0x7efbf40f0ce0 msgr2=0x7efbf40f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:36.020 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.020+0000 7efbfb7a7700 1 -- 172.21.15.58:0/1816171822 shutdown_connections 2021-11-20T05:37:36.021 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.020+0000 7efbfb7a7700 1 -- 172.21.15.58:0/1816171822 wait complete. 2021-11-20T05:37:36.021 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.021+0000 7efbfb7a7700 1 Processor -- start 2021-11-20T05:37:36.021 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.021+0000 7efbfb7a7700 1 -- start start 2021-11-20T05:37:36.021 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.021+0000 7efbfb7a7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbf40f5580 0x7efbf4105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:36.022 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.021+0000 7efbfb7a7700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbf40f6820 0x7efbf4106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:36.022 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.021+0000 7efbfb7a7700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf4106890 con 0x7efbf40f6820 2021-11-20T05:37:36.022 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.021+0000 7efbfb7a7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf41069d0 con 0x7efbf40f5580 2021-11-20T05:37:36.022 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.021+0000 7efbf8d42700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbf40f6820 0x7efbf4106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:36.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.021+0000 7efbf8d42700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbf40f6820 0x7efbf4106290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41736/0 (socket says 172.21.15.58:41736) 2021-11-20T05:37:36.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.021+0000 7efbf8d42700 1 -- 172.21.15.58:0/3080808563 learned_addr learned my addr 172.21.15.58:0/3080808563 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:36.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.022+0000 7efbf9543700 1 --2- 172.21.15.58:0/3080808563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbf40f5580 0x7efbf4105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:36.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.022+0000 7efbf8d42700 1 -- 172.21.15.58:0/3080808563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbf40f5580 msgr2=0x7efbf4105da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:36.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.022+0000 7efbf8d42700 1 --2- 172.21.15.58:0/3080808563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbf40f5580 0x7efbf4105da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:36.024 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.022+0000 7efbf8d42700 1 -- 172.21.15.58:0/3080808563 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf0005040 con 0x7efbf40f6820 2021-11-20T05:37:36.024 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.022+0000 7efbf9543700 1 --2- 172.21.15.58:0/3080808563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbf40f5580 0x7efbf4105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:37:36.024 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.022+0000 7efbf8d42700 1 --2- 172.21.15.58:0/3080808563 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbf40f6820 0x7efbf4106290 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7efbf0009f50 tx=0x7efbf0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:36.024 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.022+0000 7efbe9ffb700 1 -- 172.21.15.58:0/3080808563 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efbf0018070 con 0x7efbf40f6820 2021-11-20T05:37:36.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.022+0000 7efbe9ffb700 1 -- 172.21.15.58:0/3080808563 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efbf0013410 con 0x7efbf40f6820 2021-11-20T05:37:36.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.022+0000 7efbfb7a7700 1 -- 172.21.15.58:0/3080808563 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf410b380 con 0x7efbf40f6820 2021-11-20T05:37:36.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.022+0000 7efbe9ffb700 1 -- 172.21.15.58:0/3080808563 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efbf001c3c0 con 0x7efbf40f6820 2021-11-20T05:37:36.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.022+0000 7efbfb7a7700 1 -- 172.21.15.58:0/3080808563 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf410b8e0 con 0x7efbf40f6820 2021-11-20T05:37:36.026 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.023+0000 7efbfb7a7700 1 -- 172.21.15.58:0/3080808563 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf4027a00 con 0x7efbf40f6820 2021-11-20T05:37:36.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.024+0000 7efbe9ffb700 1 -- 172.21.15.58:0/3080808563 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7efbf001d420 con 0x7efbf40f6820 2021-11-20T05:37:36.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.024+0000 7efbe9ffb700 1 --2- 172.21.15.58:0/3080808563 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efbe005c2a0 0x7efbe005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:36.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.024+0000 7efbe9ffb700 1 -- 172.21.15.58:0/3080808563 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7efbf0077f70 con 0x7efbf40f6820 2021-11-20T05:37:36.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.024+0000 7efbf9543700 1 --2- 172.21.15.58:0/3080808563 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efbe005c2a0 0x7efbe005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:36.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.024+0000 7efbf9543700 1 --2- 172.21.15.58:0/3080808563 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efbe005c2a0 0x7efbe005e760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7efbf4107290 tx=0x7efbe4009c00).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:36.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.028+0000 7efbe9ffb700 1 -- 172.21.15.58:0/3080808563 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efbf00a42b0 con 0x7efbf40f6820 2021-11-20T05:37:36.222 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.225+0000 7efbfb7a7700 1 -- 172.21.15.58:0/3080808563 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efbf40fdd10 con 0x7efbf40f6820 2021-11-20T05:37:36.223 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.226+0000 7efbe9ffb700 1 -- 172.21.15.58:0/3080808563 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2576 (secure 0 0 0) 0x7efbf004a880 con 0x7efbf40f6820 2021-11-20T05:37:36.225 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:36.225 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:27.270544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24371,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24371":{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:36.228 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.227+0000 7efbfb7a7700 1 -- 172.21.15.58:0/3080808563 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efbe005c2a0 msgr2=0x7efbe005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:36.229 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.227+0000 7efbfb7a7700 1 --2- 172.21.15.58:0/3080808563 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efbe005c2a0 0x7efbe005e760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7efbf4107290 tx=0x7efbe4009c00).stop 2021-11-20T05:37:36.229 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.227+0000 7efbfb7a7700 1 -- 172.21.15.58:0/3080808563 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbf40f6820 msgr2=0x7efbf4106290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:36.230 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.227+0000 7efbfb7a7700 1 --2- 172.21.15.58:0/3080808563 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbf40f6820 0x7efbf4106290 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7efbf0009f50 tx=0x7efbf0009b30).stop 2021-11-20T05:37:36.230 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.227+0000 7efbfb7a7700 1 -- 172.21.15.58:0/3080808563 shutdown_connections 2021-11-20T05:37:36.231 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.227+0000 7efbfb7a7700 1 --2- 172.21.15.58:0/3080808563 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efbe005c2a0 0x7efbe005e760 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:36.231 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.227+0000 7efbfb7a7700 1 --2- 172.21.15.58:0/3080808563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbf40f5580 0x7efbf4105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:36.232 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.227+0000 7efbfb7a7700 1 --2- 172.21.15.58:0/3080808563 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbf40f6820 0x7efbf4106290 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:36.233 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.227+0000 7efbfb7a7700 1 -- 172.21.15.58:0/3080808563 >> 172.21.15.58:0/3080808563 conn(0x7efbf40f0ce0 msgr2=0x7efbf40fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:36.234 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.227+0000 7efbfb7a7700 1 -- 172.21.15.58:0/3080808563 shutdown_connections 2021-11-20T05:37:36.234 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:36.227+0000 7efbfb7a7700 1 -- 172.21.15.58:0/3080808563 wait complete. 2021-11-20T05:37:36.235 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 72 2021-11-20T05:37:36.237 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:27.270544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24371, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24371': {'gid': 24371, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 18, 'addr': '172.21.15.187:6827/2249219455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 2249219455}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 2249219455}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:36.237 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24371:up:rejoin 2021-11-20T05:37:36.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:36 smithi187 conmon[36358]: audit 2021-11-20T05:37:36.226863+0000 mon.smithi058 (mon.0) 757 : audit [DBG] from='client.? 172.21.15.58:0/3080808563' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:36.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:36 smithi058 conmon[33537]: audit 2021-11-20T05:37:36.226863+0000 2021-11-20T05:37:36.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:36 smithi058 conmon[33537]: mon.smithi058 (mon.0) 757 : audit [DBG] from='client.? 172.21.15.58:0/3080808563' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:37.238 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:37.372 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.375+0000 7f4cbe041700 1 -- 172.21.15.58:0/1175268304 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4cb80f6840 msgr2=0x7f4cb80f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:37.373 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.375+0000 7f4cbe041700 1 --2- 172.21.15.58:0/1175268304 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4cb80f6840 0x7f4cb80f6ca0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f4ca0004660 tx=0x7f4ca0009b30).stop 2021-11-20T05:37:37.373 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.375+0000 7f4cbe041700 1 -- 172.21.15.58:0/1175268304 shutdown_connections 2021-11-20T05:37:37.373 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.375+0000 7f4cbe041700 1 --2- 172.21.15.58:0/1175268304 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4cb80f6840 0x7f4cb80f6ca0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:37.373 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.375+0000 7f4cbe041700 1 --2- 172.21.15.58:0/1175268304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4cb80f55a0 0x7f4cb80f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:37.374 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.375+0000 7f4cbe041700 1 -- 172.21.15.58:0/1175268304 >> 172.21.15.58:0/1175268304 conn(0x7f4cb80f0ce0 msgr2=0x7f4cb80f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:37.374 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.375+0000 7f4cbe041700 1 -- 172.21.15.58:0/1175268304 shutdown_connections 2021-11-20T05:37:37.374 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.375+0000 7f4cbe041700 1 -- 172.21.15.58:0/1175268304 wait complete. 2021-11-20T05:37:37.374 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.375+0000 7f4cbe041700 1 Processor -- start 2021-11-20T05:37:37.375 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.375+0000 7f4cbe041700 1 -- start start 2021-11-20T05:37:37.375 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.376+0000 7f4cbe041700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4cb80f55a0 0x7f4cb805ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:37.375 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.376+0000 7f4cbe041700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4cb80f6840 0x7f4cb810bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:37.375 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.376+0000 7f4cbe041700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb810c180 con 0x7f4cb80f6840 2021-11-20T05:37:37.376 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.376+0000 7f4cbe041700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb810c2c0 con 0x7f4cb80f55a0 2021-11-20T05:37:37.376 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.376+0000 7f4caffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4cb80f6840 0x7f4cb810bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:37.376 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.376+0000 7f4caffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4cb80f6840 0x7f4cb810bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41748/0 (socket says 172.21.15.58:41748) 2021-11-20T05:37:37.376 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.376+0000 7f4caffff700 1 -- 172.21.15.58:0/521224978 learned_addr learned my addr 172.21.15.58:0/521224978 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:37.376 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.376+0000 7f4cb77fe700 1 --2- 172.21.15.58:0/521224978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4cb80f55a0 0x7f4cb805ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:37.377 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.376+0000 7f4caffff700 1 -- 172.21.15.58:0/521224978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4cb80f55a0 msgr2=0x7f4cb805ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:37.377 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.376+0000 7f4caffff700 1 --2- 172.21.15.58:0/521224978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4cb80f55a0 0x7f4cb805ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:37.377 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.376+0000 7f4caffff700 1 -- 172.21.15.58:0/521224978 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca0005040 con 0x7f4cb80f6840 2021-11-20T05:37:37.377 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.376+0000 7f4caffff700 1 --2- 172.21.15.58:0/521224978 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4cb80f6840 0x7f4cb810bc90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f4ca0009830 tx=0x7f4ca0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:37.379 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.377+0000 7f4cb4ff9700 1 -- 172.21.15.58:0/521224978 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ca0018070 con 0x7f4cb80f6840 2021-11-20T05:37:37.379 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.377+0000 7f4cb4ff9700 1 -- 172.21.15.58:0/521224978 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ca0013410 con 0x7f4cb80f6840 2021-11-20T05:37:37.380 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.377+0000 7f4cb4ff9700 1 -- 172.21.15.58:0/521224978 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ca001c3c0 con 0x7f4cb80f6840 2021-11-20T05:37:37.380 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.377+0000 7f4cbe041700 1 -- 172.21.15.58:0/521224978 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb810c4c0 con 0x7f4cb80f6840 2021-11-20T05:37:37.380 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.377+0000 7f4cbe041700 1 -- 172.21.15.58:0/521224978 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb810ca20 con 0x7f4cb80f6840 2021-11-20T05:37:37.380 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.378+0000 7f4cbe041700 1 -- 172.21.15.58:0/521224978 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb8027a00 con 0x7f4cb80f6840 2021-11-20T05:37:37.381 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.378+0000 7f4cb4ff9700 1 -- 172.21.15.58:0/521224978 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4ca001d420 con 0x7f4cb80f6840 2021-11-20T05:37:37.381 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.379+0000 7f4cb4ff9700 1 --2- 172.21.15.58:0/521224978 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4c9805c250 0x7f4c9805e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:37.381 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.379+0000 7f4cb4ff9700 1 -- 172.21.15.58:0/521224978 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f4ca0077e20 con 0x7f4cb80f6840 2021-11-20T05:37:37.381 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.379+0000 7f4cb77fe700 1 --2- 172.21.15.58:0/521224978 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4c9805c250 0x7f4c9805e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:37.382 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.379+0000 7f4cb77fe700 1 --2- 172.21.15.58:0/521224978 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4c9805c250 0x7f4c9805e710 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f4cb80f66a0 tx=0x7f4ca800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:37.382 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.383+0000 7f4cb4ff9700 1 -- 172.21.15.58:0/521224978 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ca004a7b0 con 0x7f4cb80f6840 2021-11-20T05:37:37.576 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.579+0000 7f4cbe041700 1 -- 172.21.15.58:0/521224978 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cb80fdd30 con 0x7f4cb80f6840 2021-11-20T05:37:37.577 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.580+0000 7f4cb4ff9700 1 -- 172.21.15.58:0/521224978 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2576 (secure 0 0 0) 0x7f4ca004ddd0 con 0x7f4cb80f6840 2021-11-20T05:37:37.577 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:37.578 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:27.270544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24371,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24371":{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:37.580 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.582+0000 7f4cbe041700 1 -- 172.21.15.58:0/521224978 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4c9805c250 msgr2=0x7f4c9805e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:37.580 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.582+0000 7f4cbe041700 1 --2- 172.21.15.58:0/521224978 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4c9805c250 0x7f4c9805e710 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f4cb80f66a0 tx=0x7f4ca800b040).stop 2021-11-20T05:37:37.580 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.582+0000 7f4cbe041700 1 -- 172.21.15.58:0/521224978 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4cb80f6840 msgr2=0x7f4cb810bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:37.581 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.582+0000 7f4cbe041700 1 --2- 172.21.15.58:0/521224978 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4cb80f6840 0x7f4cb810bc90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f4ca0009830 tx=0x7f4ca0009b30).stop 2021-11-20T05:37:37.581 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.582+0000 7f4cbe041700 1 -- 172.21.15.58:0/521224978 shutdown_connections 2021-11-20T05:37:37.581 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.582+0000 7f4cbe041700 1 --2- 172.21.15.58:0/521224978 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4c9805c250 0x7f4c9805e710 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:37.582 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.582+0000 7f4cbe041700 1 --2- 172.21.15.58:0/521224978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4cb80f55a0 0x7f4cb805ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:37.582 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.582+0000 7f4cbe041700 1 --2- 172.21.15.58:0/521224978 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4cb80f6840 0x7f4cb810bc90 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:37.582 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.582+0000 7f4cbe041700 1 -- 172.21.15.58:0/521224978 >> 172.21.15.58:0/521224978 conn(0x7f4cb80f0ce0 msgr2=0x7f4cb80fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:37.582 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.582+0000 7f4cbe041700 1 -- 172.21.15.58:0/521224978 shutdown_connections 2021-11-20T05:37:37.583 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:37.582+0000 7f4cbe041700 1 -- 172.21.15.58:0/521224978 wait complete. 2021-11-20T05:37:37.583 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 72 2021-11-20T05:37:37.589 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:37 smithi058 conmon[33537]: cluster 2021-11-20T05:37:36.286813+0000 mgr.smithi058.zagdlk 2021-11-20T05:37:37.589 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:37 smithi058 conmon[33537]: (mgr.14164) 297 : cluster [DBG] pgmap v270: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:37:37.590 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:27.270544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24371, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24371': {'gid': 24371, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 18, 'addr': '172.21.15.187:6827/2249219455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 2249219455}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 2249219455}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:37.591 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24371:up:rejoin 2021-11-20T05:37:37.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:37 smithi187 conmon[36358]: cluster 2021-11-20T05:37:36.286813+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:37:37.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:37 smithi187 conmon[36358]: mgr.14164) 297 : cluster [DBG] pgmap v270: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:37:38.592 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:38.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:38 smithi187 conmon[36358]: audit 2021-11-20T05:37:37.580972+0000 mon.smithi058 (mon.0 2021-11-20T05:37:38.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:38 smithi187 conmon[36358]: ) 758 : audit [DBG] from='client.? 172.21.15.58:0/521224978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:38.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:38 smithi058 conmon[33537]: audit 2021-11-20T05:37:37.580972+0000 mon.smithi058 (mon.0) 758 : 2021-11-20T05:37:38.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:38 smithi058 conmon[33537]: audit [DBG] from='client.? 172.21.15.58:0/521224978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:38.725 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.728+0000 7f70453d9700 1 -- 172.21.15.58:0/3751200483 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f6770 msgr2=0x7f70400f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:38.726 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.728+0000 7f70453d9700 1 --2- 172.21.15.58:0/3751200483 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f6770 0x7f70400f6bd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f7034004660 tx=0x7f7034009b30).stop 2021-11-20T05:37:38.726 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.728+0000 7f70453d9700 1 -- 172.21.15.58:0/3751200483 shutdown_connections 2021-11-20T05:37:38.726 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.728+0000 7f70453d9700 1 --2- 172.21.15.58:0/3751200483 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f6770 0x7f70400f6bd0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:38.726 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.728+0000 7f70453d9700 1 --2- 172.21.15.58:0/3751200483 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70400f54d0 0x7f70400f58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:38.727 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.728+0000 7f70453d9700 1 -- 172.21.15.58:0/3751200483 >> 172.21.15.58:0/3751200483 conn(0x7f70400f0c10 msgr2=0x7f70400f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:38.727 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.728+0000 7f70453d9700 1 -- 172.21.15.58:0/3751200483 shutdown_connections 2021-11-20T05:37:38.727 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.728+0000 7f70453d9700 1 -- 172.21.15.58:0/3751200483 wait complete. 2021-11-20T05:37:38.727 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.729+0000 7f70453d9700 1 Processor -- start 2021-11-20T05:37:38.727 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.729+0000 7f70453d9700 1 -- start start 2021-11-20T05:37:38.730 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.729+0000 7f70453d9700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f54d0 0x7f7040108ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:38.730 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.730+0000 7f70453d9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70400f6770 0x7f70401093b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:38.730 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.730+0000 7f70453d9700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7040109960 con 0x7f70400f54d0 2021-11-20T05:37:38.731 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.730+0000 7f70453d9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7040109aa0 con 0x7f70400f6770 2021-11-20T05:37:38.731 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.730+0000 7f703ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f54d0 0x7f7040108ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:38.731 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.730+0000 7f703ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f54d0 0x7f7040108ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41756/0 (socket says 172.21.15.58:41756) 2021-11-20T05:37:38.731 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.730+0000 7f703ffff700 1 -- 172.21.15.58:0/96565794 learned_addr learned my addr 172.21.15.58:0/96565794 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:38.732 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.730+0000 7f703f7fe700 1 --2- 172.21.15.58:0/96565794 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70400f6770 0x7f70401093b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:38.732 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.730+0000 7f703ffff700 1 -- 172.21.15.58:0/96565794 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70400f6770 msgr2=0x7f70401093b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:38.732 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.730+0000 7f703ffff700 1 --2- 172.21.15.58:0/96565794 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70400f6770 0x7f70401093b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:38.732 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.730+0000 7f703ffff700 1 -- 172.21.15.58:0/96565794 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7034005040 con 0x7f70400f54d0 2021-11-20T05:37:38.732 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.730+0000 7f703ffff700 1 --2- 172.21.15.58:0/96565794 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f54d0 0x7f7040108ec0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f7030002700 tx=0x7f7030009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:38.733 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.730+0000 7f703cff9700 1 -- 172.21.15.58:0/96565794 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70300079c0 con 0x7f70400f54d0 2021-11-20T05:37:38.733 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.730+0000 7f70453d9700 1 -- 172.21.15.58:0/96565794 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f704010a440 con 0x7f70400f54d0 2021-11-20T05:37:38.733 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.730+0000 7f703cff9700 1 -- 172.21.15.58:0/96565794 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f703000f040 con 0x7f70400f54d0 2021-11-20T05:37:38.733 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.730+0000 7f703cff9700 1 -- 172.21.15.58:0/96565794 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7030014850 con 0x7f70400f54d0 2021-11-20T05:37:38.734 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.730+0000 7f70453d9700 1 -- 172.21.15.58:0/96565794 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f704010aa00 con 0x7f70400f54d0 2021-11-20T05:37:38.734 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.731+0000 7f70453d9700 1 -- 172.21.15.58:0/96565794 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7040045ad0 con 0x7f70400f54d0 2021-11-20T05:37:38.734 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.732+0000 7f703cff9700 1 -- 172.21.15.58:0/96565794 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f703001b060 con 0x7f70400f54d0 2021-11-20T05:37:38.734 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.732+0000 7f703cff9700 1 --2- 172.21.15.58:0/96565794 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f702805c300 0x7f702805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:38.735 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.732+0000 7f703cff9700 1 -- 172.21.15.58:0/96565794 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7030076190 con 0x7f70400f54d0 2021-11-20T05:37:38.735 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.732+0000 7f703f7fe700 1 --2- 172.21.15.58:0/96565794 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f702805c300 0x7f702805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:38.735 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.733+0000 7f703f7fe700 1 --2- 172.21.15.58:0/96565794 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f702805c300 0x7f702805e7c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f7034009f50 tx=0x7f7034009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:38.736 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.736+0000 7f703cff9700 1 -- 172.21.15.58:0/96565794 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7030048aa0 con 0x7f70400f54d0 2021-11-20T05:37:38.928 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.932+0000 7f70453d9700 1 -- 172.21.15.58:0/96565794 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70400fdca0 con 0x7f70400f54d0 2021-11-20T05:37:38.929 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.932+0000 7f703cff9700 1 -- 172.21.15.58:0/96565794 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2576 (secure 0 0 0) 0x7f703004c0c0 con 0x7f70400f54d0 2021-11-20T05:37:38.929 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:38.929 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:27.270544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24371,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24371":{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:38.931 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.934+0000 7f70453d9700 1 -- 172.21.15.58:0/96565794 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f702805c300 msgr2=0x7f702805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:38.931 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.934+0000 7f70453d9700 1 --2- 172.21.15.58:0/96565794 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f702805c300 0x7f702805e7c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f7034009f50 tx=0x7f7034009b30).stop 2021-11-20T05:37:38.931 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.934+0000 7f70453d9700 1 -- 172.21.15.58:0/96565794 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f54d0 msgr2=0x7f7040108ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:38.932 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.934+0000 7f70453d9700 1 --2- 172.21.15.58:0/96565794 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f54d0 0x7f7040108ec0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f7030002700 tx=0x7f7030009e00).stop 2021-11-20T05:37:38.932 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.934+0000 7f70453d9700 1 -- 172.21.15.58:0/96565794 shutdown_connections 2021-11-20T05:37:38.932 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.934+0000 7f70453d9700 1 --2- 172.21.15.58:0/96565794 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f702805c300 0x7f702805e7c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:38.932 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.934+0000 7f70453d9700 1 --2- 172.21.15.58:0/96565794 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f70400f54d0 0x7f7040108ec0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:38.933 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.934+0000 7f70453d9700 1 --2- 172.21.15.58:0/96565794 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70400f6770 0x7f70401093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:38.933 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.934+0000 7f70453d9700 1 -- 172.21.15.58:0/96565794 >> 172.21.15.58:0/96565794 conn(0x7f70400f0c10 msgr2=0x7f704005f760 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:38.933 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.934+0000 7f70453d9700 1 -- 172.21.15.58:0/96565794 shutdown_connections 2021-11-20T05:37:38.933 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:38.934+0000 7f70453d9700 1 -- 172.21.15.58:0/96565794 wait complete. 2021-11-20T05:37:38.933 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 72 2021-11-20T05:37:38.943 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:27.270544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24371, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24371': {'gid': 24371, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 18, 'addr': '172.21.15.187:6827/2249219455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 2249219455}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 2249219455}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:38.944 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24371:up:rejoin 2021-11-20T05:37:39.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:39 smithi187 conmon[36358]: cluster 2021-11-20T05:37:38.287305+0000 mgr.smithi058.zagdlk (mgr.14164) 298 : 2021-11-20T05:37:39.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:39 smithi187 conmon[36358]: cluster [DBG] pgmap v271: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:37:39.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:39 smithi187 conmon[36358]: audit 2021-11-20T05:37:38.933424+0000 mon.smithi058 (mon.0) 759 : audit [DBG] from='client.? 172.21.15.58:0/96565794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:39.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:39 smithi058 conmon[33537]: cluster 2021-11-20T05:37:38.287305+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:37:39.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:39 smithi058 conmon[33537]: 14164) 298 : cluster [DBG] pgmap v271: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:37:39.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:39 smithi058 conmon[33537]: audit 2021-11-20T05:37:38.933424+0000 mon.smithi058 (mon.0) 759 : 2021-11-20T05:37:39.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:39 smithi058 conmon[33537]: audit [DBG] from='client.? 172.21.15.58:0/96565794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:39.945 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:40.081 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.081+0000 7fea153bb700 1 -- 172.21.15.58:0/4165509446 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fea100f6a50 msgr2=0x7fea100f6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:40.081 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.081+0000 7fea153bb700 1 --2- 172.21.15.58:0/4165509446 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fea100f6a50 0x7fea100f6e70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fea00004660 tx=0x7fea00009b30).stop 2021-11-20T05:37:40.081 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.082+0000 7fea153bb700 1 -- 172.21.15.58:0/4165509446 shutdown_connections 2021-11-20T05:37:40.082 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.082+0000 7fea153bb700 1 --2- 172.21.15.58:0/4165509446 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea100f7360 0x7fea100f57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:40.082 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.082+0000 7fea153bb700 1 --2- 172.21.15.58:0/4165509446 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fea100f6a50 0x7fea100f6e70 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:40.082 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.082+0000 7fea153bb700 1 -- 172.21.15.58:0/4165509446 >> 172.21.15.58:0/4165509446 conn(0x7fea100f0400 msgr2=0x7fea100f2820 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:40.082 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.082+0000 7fea153bb700 1 -- 172.21.15.58:0/4165509446 shutdown_connections 2021-11-20T05:37:40.082 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.082+0000 7fea153bb700 1 -- 172.21.15.58:0/4165509446 wait complete. 2021-11-20T05:37:40.083 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.082+0000 7fea153bb700 1 Processor -- start 2021-11-20T05:37:40.083 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.082+0000 7fea153bb700 1 -- start start 2021-11-20T05:37:40.083 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.082+0000 7fea153bb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea100f6a50 0x7fea10109a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:40.083 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.083+0000 7fea153bb700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fea100f7360 0x7fea1010bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:40.084 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.083+0000 7fea153bb700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea100f54a0 con 0x7fea100f7360 2021-11-20T05:37:40.084 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.083+0000 7fea153bb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea100f55e0 con 0x7fea100f6a50 2021-11-20T05:37:40.084 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.083+0000 7fea0f7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fea100f7360 0x7fea1010bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:40.084 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.083+0000 7fea0f7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fea100f7360 0x7fea1010bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41770/0 (socket says 172.21.15.58:41770) 2021-11-20T05:37:40.085 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.083+0000 7fea0f7fe700 1 -- 172.21.15.58:0/2065018583 learned_addr learned my addr 172.21.15.58:0/2065018583 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:40.085 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.083+0000 7fea0ffff700 1 --2- 172.21.15.58:0/2065018583 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea100f6a50 0x7fea10109a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:40.086 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.083+0000 7fea0f7fe700 1 -- 172.21.15.58:0/2065018583 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea100f6a50 msgr2=0x7fea10109a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:40.087 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.083+0000 7fea0f7fe700 1 --2- 172.21.15.58:0/2065018583 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea100f6a50 0x7fea10109a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:40.087 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.083+0000 7fea0f7fe700 1 -- 172.21.15.58:0/2065018583 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea00005040 con 0x7fea100f7360 2021-11-20T05:37:40.087 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.083+0000 7fea0f7fe700 1 --2- 172.21.15.58:0/2065018583 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fea100f7360 0x7fea1010bf50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fea040026d0 tx=0x7fea04009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:40.087 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.083+0000 7fea0cff9700 1 -- 172.21.15.58:0/2065018583 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea040079c0 con 0x7fea100f7360 2021-11-20T05:37:40.088 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.083+0000 7fea0cff9700 1 -- 172.21.15.58:0/2065018583 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea0400f040 con 0x7fea100f7360 2021-11-20T05:37:40.088 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.083+0000 7fea153bb700 1 -- 172.21.15.58:0/2065018583 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea1010c500 con 0x7fea100f7360 2021-11-20T05:37:40.088 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.083+0000 7fea0cff9700 1 -- 172.21.15.58:0/2065018583 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea040148b0 con 0x7fea100f7360 2021-11-20T05:37:40.089 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.083+0000 7fea153bb700 1 -- 172.21.15.58:0/2065018583 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1010cac0 con 0x7fea100f7360 2021-11-20T05:37:40.089 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.084+0000 7fea153bb700 1 -- 172.21.15.58:0/2065018583 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea10045ad0 con 0x7fea100f7360 2021-11-20T05:37:40.089 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.085+0000 7fea0cff9700 1 -- 172.21.15.58:0/2065018583 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fea04007b20 con 0x7fea100f7360 2021-11-20T05:37:40.090 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.085+0000 7fea0cff9700 1 --2- 172.21.15.58:0/2065018583 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe9f805c300 0x7fe9f805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:40.091 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.085+0000 7fea0cff9700 1 -- 172.21.15.58:0/2065018583 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fea040761f0 con 0x7fea100f7360 2021-11-20T05:37:40.091 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.085+0000 7fea0ffff700 1 --2- 172.21.15.58:0/2065018583 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe9f805c300 0x7fe9f805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:40.092 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.086+0000 7fea0ffff700 1 --2- 172.21.15.58:0/2065018583 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe9f805c300 0x7fe9f805e7c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fea100f5120 tx=0x7fea00009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:40.092 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.089+0000 7fea0cff9700 1 -- 172.21.15.58:0/2065018583 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea04048c30 con 0x7fea100f7360 2021-11-20T05:37:40.284 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.288+0000 7fea153bb700 1 -- 172.21.15.58:0/2065018583 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea10027a00 con 0x7fea100f7360 2021-11-20T05:37:40.285 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.288+0000 7fea0cff9700 1 -- 172.21.15.58:0/2065018583 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2576 (secure 0 0 0) 0x7fea0404c250 con 0x7fea100f7360 2021-11-20T05:37:40.285 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:40.286 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:27.270544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24371,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24371":{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:40.287 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.290+0000 7fea153bb700 1 -- 172.21.15.58:0/2065018583 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe9f805c300 msgr2=0x7fe9f805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:40.287 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.290+0000 7fea153bb700 1 --2- 172.21.15.58:0/2065018583 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe9f805c300 0x7fe9f805e7c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fea100f5120 tx=0x7fea00009b30).stop 2021-11-20T05:37:40.287 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.290+0000 7fea153bb700 1 -- 172.21.15.58:0/2065018583 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fea100f7360 msgr2=0x7fea1010bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:40.288 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.290+0000 7fea153bb700 1 --2- 172.21.15.58:0/2065018583 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fea100f7360 0x7fea1010bf50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fea040026d0 tx=0x7fea04009de0).stop 2021-11-20T05:37:40.288 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.290+0000 7fea153bb700 1 -- 172.21.15.58:0/2065018583 shutdown_connections 2021-11-20T05:37:40.288 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.290+0000 7fea153bb700 1 --2- 172.21.15.58:0/2065018583 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe9f805c300 0x7fe9f805e7c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:40.288 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.290+0000 7fea153bb700 1 --2- 172.21.15.58:0/2065018583 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fea100f6a50 0x7fea10109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:40.289 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.290+0000 7fea153bb700 1 --2- 172.21.15.58:0/2065018583 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fea100f7360 0x7fea1010bf50 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:40.289 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.290+0000 7fea153bb700 1 -- 172.21.15.58:0/2065018583 >> 172.21.15.58:0/2065018583 conn(0x7fea100f0400 msgr2=0x7fea100fa250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:40.289 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.290+0000 7fea153bb700 1 -- 172.21.15.58:0/2065018583 shutdown_connections 2021-11-20T05:37:40.289 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:40.290+0000 7fea153bb700 1 -- 172.21.15.58:0/2065018583 wait complete. 2021-11-20T05:37:40.289 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 72 2021-11-20T05:37:40.299 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:27.270544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24371, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24371': {'gid': 24371, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 18, 'addr': '172.21.15.187:6827/2249219455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 2249219455}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 2249219455}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:40.299 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24371:up:rejoin 2021-11-20T05:37:40.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:40 smithi187 conmon[36358]: audit 2021-11-20T05:37:40.289417+0000 mon.smithi058 (mon.0) 760 : audit [DBG] from='client.? 172.21.15.58:0/2065018583' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:40.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:40 smithi058 conmon[33537]: audit 2021-11-20T05:37:40.289417+0000 mon.smithi058 (mon.0) 760 : 2021-11-20T05:37:40.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:40 smithi058 conmon[33537]: audit [DBG] from='client.? 172.21.15.58:0/2065018583' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:41.300 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:41.434 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.436+0000 7f361a138700 1 -- 172.21.15.58:0/2981582260 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f36140f6820 msgr2=0x7f36140f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:41.435 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.436+0000 7f361a138700 1 --2- 172.21.15.58:0/2981582260 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f36140f6820 0x7f36140f6c80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f3608004660 tx=0x7f3608009b30).stop 2021-11-20T05:37:41.435 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.436+0000 7f361a138700 1 -- 172.21.15.58:0/2981582260 shutdown_connections 2021-11-20T05:37:41.435 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.436+0000 7f361a138700 1 --2- 172.21.15.58:0/2981582260 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f36140f6820 0x7f36140f6c80 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:41.436 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.436+0000 7f361a138700 1 --2- 172.21.15.58:0/2981582260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36140f5580 0x7f36140f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:41.436 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.436+0000 7f361a138700 1 -- 172.21.15.58:0/2981582260 >> 172.21.15.58:0/2981582260 conn(0x7f36140f0ce0 msgr2=0x7f36140f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:41.436 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.436+0000 7f361a138700 1 -- 172.21.15.58:0/2981582260 shutdown_connections 2021-11-20T05:37:41.436 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.436+0000 7f361a138700 1 -- 172.21.15.58:0/2981582260 wait complete. 2021-11-20T05:37:41.436 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.437+0000 7f361a138700 1 Processor -- start 2021-11-20T05:37:41.437 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.437+0000 7f361a138700 1 -- start start 2021-11-20T05:37:41.437 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.437+0000 7f361a138700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f36140f5580 0x7f3614060470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:41.437 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.437+0000 7f361a138700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36140f6820 0x7f361410e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:41.437 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.437+0000 7f361a138700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3614107a70 con 0x7f36140f5580 2021-11-20T05:37:41.438 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.437+0000 7f361a138700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361410e580 con 0x7f36140f6820 2021-11-20T05:37:41.438 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.437+0000 7f36137fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f36140f5580 0x7f3614060470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:41.438 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.437+0000 7f36137fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f36140f5580 0x7f3614060470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41778/0 (socket says 172.21.15.58:41778) 2021-11-20T05:37:41.438 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.437+0000 7f36137fe700 1 -- 172.21.15.58:0/4190770032 learned_addr learned my addr 172.21.15.58:0/4190770032 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:41.438 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.437+0000 7f3612ffd700 1 --2- 172.21.15.58:0/4190770032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36140f6820 0x7f361410e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:41.439 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.437+0000 7f36137fe700 1 -- 172.21.15.58:0/4190770032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36140f6820 msgr2=0x7f361410e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:41.439 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.437+0000 7f36137fe700 1 --2- 172.21.15.58:0/4190770032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36140f6820 0x7f361410e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:41.439 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.437+0000 7f36137fe700 1 -- 172.21.15.58:0/4190770032 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3608005040 con 0x7f36140f5580 2021-11-20T05:37:41.439 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.437+0000 7f3612ffd700 1 --2- 172.21.15.58:0/4190770032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36140f6820 0x7f361410e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:37:41.440 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.438+0000 7f36137fe700 1 --2- 172.21.15.58:0/4190770032 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f36140f5580 0x7f3614060470 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f360400c530 tx=0x7f3604007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:41.440 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.438+0000 7f35fbfff700 1 -- 172.21.15.58:0/4190770032 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f360400f650 con 0x7f36140f5580 2021-11-20T05:37:41.440 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.438+0000 7f361a138700 1 -- 172.21.15.58:0/4190770032 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361410e720 con 0x7f36140f5580 2021-11-20T05:37:41.440 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.438+0000 7f35fbfff700 1 -- 172.21.15.58:0/4190770032 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f360400fc60 con 0x7f36140f5580 2021-11-20T05:37:41.440 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.438+0000 7f35fbfff700 1 -- 172.21.15.58:0/4190770032 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3604012d20 con 0x7f36140f5580 2021-11-20T05:37:41.441 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.438+0000 7f361a138700 1 -- 172.21.15.58:0/4190770032 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f361410ecb0 con 0x7f36140f5580 2021-11-20T05:37:41.442 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.439+0000 7f361a138700 1 -- 172.21.15.58:0/4190770032 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3614046330 con 0x7f36140f5580 2021-11-20T05:37:41.442 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.439+0000 7f35fbfff700 1 -- 172.21.15.58:0/4190770032 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3604019030 con 0x7f36140f5580 2021-11-20T05:37:41.442 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.440+0000 7f35fbfff700 1 --2- 172.21.15.58:0/4190770032 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f35fc05c2a0 0x7f35fc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:41.443 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.440+0000 7f35fbfff700 1 -- 172.21.15.58:0/4190770032 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3604075ee0 con 0x7f36140f5580 2021-11-20T05:37:41.443 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.440+0000 7f3612ffd700 1 --2- 172.21.15.58:0/4190770032 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f35fc05c2a0 0x7f35fc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:41.443 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.440+0000 7f3612ffd700 1 --2- 172.21.15.58:0/4190770032 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f35fc05c2a0 0x7f35fc05e760 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f3608009d60 tx=0x7f3608009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:41.443 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.444+0000 7f35fbfff700 1 -- 172.21.15.58:0/4190770032 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3604048840 con 0x7f36140f5580 2021-11-20T05:37:41.637 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.640+0000 7f361a138700 1 -- 172.21.15.58:0/4190770032 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36140fdd10 con 0x7f36140f5580 2021-11-20T05:37:41.637 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.641+0000 7f35fbfff700 1 -- 172.21.15.58:0/4190770032 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2576 (secure 0 0 0) 0x7f3604017090 con 0x7f36140f5580 2021-11-20T05:37:41.638 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:41.638 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:27.270544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24371,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24371":{"gid":24371,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.187:6827/2249219455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2249219455},{"type":"v1","addr":"172.21.15.187:6827","nonce":2249219455}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:41.639 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.642+0000 7f361a138700 1 -- 172.21.15.58:0/4190770032 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f35fc05c2a0 msgr2=0x7f35fc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:41.639 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.642+0000 7f361a138700 1 --2- 172.21.15.58:0/4190770032 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f35fc05c2a0 0x7f35fc05e760 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f3608009d60 tx=0x7f3608009b30).stop 2021-11-20T05:37:41.640 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.642+0000 7f361a138700 1 -- 172.21.15.58:0/4190770032 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f36140f5580 msgr2=0x7f3614060470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:41.640 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.642+0000 7f361a138700 1 --2- 172.21.15.58:0/4190770032 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f36140f5580 0x7f3614060470 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f360400c530 tx=0x7f3604007e10).stop 2021-11-20T05:37:41.640 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.642+0000 7f361a138700 1 -- 172.21.15.58:0/4190770032 shutdown_connections 2021-11-20T05:37:41.640 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.642+0000 7f361a138700 1 --2- 172.21.15.58:0/4190770032 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f35fc05c2a0 0x7f35fc05e760 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:41.641 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.642+0000 7f361a138700 1 --2- 172.21.15.58:0/4190770032 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f36140f5580 0x7f3614060470 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:41.642 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.642+0000 7f361a138700 1 --2- 172.21.15.58:0/4190770032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f36140f6820 0x7f361410e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:41.642 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.642+0000 7f361a138700 1 -- 172.21.15.58:0/4190770032 >> 172.21.15.58:0/4190770032 conn(0x7f36140f0ce0 msgr2=0x7f36140fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:41.642 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.643+0000 7f361a138700 1 -- 172.21.15.58:0/4190770032 shutdown_connections 2021-11-20T05:37:41.642 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:41.643+0000 7f361a138700 1 -- 172.21.15.58:0/4190770032 wait complete. 2021-11-20T05:37:41.643 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 72 2021-11-20T05:37:41.649 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:41 smithi058 conmon[33537]: cluster 2021-11-20T05:37:40.287814+0000 mgr.smithi058.zagdlk 2021-11-20T05:37:41.650 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:41 smithi058 conmon[33537]: (mgr.14164) 299 : cluster [DBG] pgmap v272: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2021-11-20T05:37:41.651 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:27.270544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24371, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24371': {'gid': 24371, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 18, 'addr': '172.21.15.187:6827/2249219455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 2249219455}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 2249219455}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:41.652 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24371:up:rejoin 2021-11-20T05:37:41.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:41 smithi187 conmon[36358]: cluster 2021-11-20T05:37:40.287814+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:37:41.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:41 smithi187 conmon[36358]: 299 : cluster [DBG] pgmap v272: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2021-11-20T05:37:42.654 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:42.683 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:42 smithi058 conmon[33537]: audit 2021-11-20T05:37:41.641626+0000 mon.smithi058 2021-11-20T05:37:42.684 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:42 smithi058 conmon[33537]: (mon.0) 761 : audit [DBG] from='client.? 172.21.15.58:0/4190770032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:42.684 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:42 smithi058 conmon[33537]: cluster 2021-11-20T05:37:42.299166+0000 mon.smithi058 (mon.0) 762 : cluster [INF] daemon mds.cephfs.smithi187.fikisa restarted 2021-11-20T05:37:42.685 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:42 smithi058 conmon[33537]: cluster 2021-11-20T05:37:42.303458+0000 mon.smithi058 (mon.0 2021-11-20T05:37:42.685 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:42 smithi058 conmon[33537]: ) 763 : cluster [DBG] osdmap e54: 6 total, 6 up, 6 in 2021-11-20T05:37:42.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:42 smithi187 conmon[36358]: audit 2021-11-20T05:37:41.641626+0000 mon.smithi058 (mon.0) 761 : audit [DBG] from='client.? 172.21.15.58:0/4190770032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:42.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:42 smithi187 conmon[36358]: cluster 2021-11-20T05:37:42.299166+0000 mon.smithi058 (mon.0) 762 : cluster [INF] daemon mds.cephfs.smithi187.fikisa restarted 2021-11-20T05:37:42.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:42 smithi187 conmon[36358]: cluster 2021-11-20T05:37:42.303458+0000 mon.smithi058 (mon.0) 763 : cluster [DBG] osdmap e54: 6 total, 6 up, 6 in 2021-11-20T05:37:42.789 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.791+0000 7f73d2a28700 1 -- 172.21.15.58:0/3786891116 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f73cc0f4780 msgr2=0x7f73cc0f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:42.790 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.791+0000 7f73d2a28700 1 --2- 172.21.15.58:0/3786891116 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f73cc0f4780 0x7f73cc0f4c00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f73c0004660 tx=0x7f73c0009b30).stop 2021-11-20T05:37:42.790 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.791+0000 7f73d2a28700 1 -- 172.21.15.58:0/3786891116 shutdown_connections 2021-11-20T05:37:42.790 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.791+0000 7f73d2a28700 1 --2- 172.21.15.58:0/3786891116 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f73cc0f4780 0x7f73cc0f4c00 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:42.790 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.791+0000 7f73d2a28700 1 --2- 172.21.15.58:0/3786891116 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73cc0f7d90 0x7f73cc0f81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:42.791 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.791+0000 7f73d2a28700 1 -- 172.21.15.58:0/3786891116 >> 172.21.15.58:0/3786891116 conn(0x7f73cc0f04a0 msgr2=0x7f73cc0f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:42.791 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.791+0000 7f73d2a28700 1 -- 172.21.15.58:0/3786891116 shutdown_connections 2021-11-20T05:37:42.791 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.791+0000 7f73d2a28700 1 -- 172.21.15.58:0/3786891116 wait complete. 2021-11-20T05:37:42.791 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.791+0000 7f73d2a28700 1 Processor -- start 2021-11-20T05:37:42.792 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.792+0000 7f73d2a28700 1 -- start start 2021-11-20T05:37:42.792 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.792+0000 7f73d2a28700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f73cc0f4780 0x7f73cc0f7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:42.792 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.792+0000 7f73d2a28700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73cc0f7d90 0x7f73cc0f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:42.792 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.792+0000 7f73d2a28700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc0f7900 con 0x7f73cc0f4780 2021-11-20T05:37:42.793 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.792+0000 7f73d2a28700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc0f5ea0 con 0x7f73cc0f7d90 2021-11-20T05:37:42.793 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.792+0000 7f73cbfff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f73cc0f4780 0x7f73cc0f7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:42.793 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.792+0000 7f73cbfff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f73cc0f4780 0x7f73cc0f7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41788/0 (socket says 172.21.15.58:41788) 2021-11-20T05:37:42.793 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.792+0000 7f73cbfff700 1 -- 172.21.15.58:0/3293062438 learned_addr learned my addr 172.21.15.58:0/3293062438 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:42.793 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.792+0000 7f73cb7fe700 1 --2- 172.21.15.58:0/3293062438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73cc0f7d90 0x7f73cc0f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:42.794 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.792+0000 7f73cbfff700 1 -- 172.21.15.58:0/3293062438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73cc0f7d90 msgr2=0x7f73cc0f59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:42.794 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.792+0000 7f73cbfff700 1 --2- 172.21.15.58:0/3293062438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73cc0f7d90 0x7f73cc0f59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:42.794 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.792+0000 7f73cbfff700 1 -- 172.21.15.58:0/3293062438 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c0005040 con 0x7f73cc0f4780 2021-11-20T05:37:42.794 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.793+0000 7f73cbfff700 1 --2- 172.21.15.58:0/3293062438 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f73cc0f4780 0x7f73cc0f7350 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f73bc002700 tx=0x7f73bc009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:42.795 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.793+0000 7f73c8ff9700 1 -- 172.21.15.58:0/3293062438 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73bc0079c0 con 0x7f73cc0f4780 2021-11-20T05:37:42.795 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.793+0000 7f73c8ff9700 1 -- 172.21.15.58:0/3293062438 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73bc00f040 con 0x7f73cc0f4780 2021-11-20T05:37:42.795 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.793+0000 7f73c8ff9700 1 -- 172.21.15.58:0/3293062438 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73bc014850 con 0x7f73cc0f4780 2021-11-20T05:37:42.795 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.793+0000 7f73d2a28700 1 -- 172.21.15.58:0/3293062438 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73cc0f6120 con 0x7f73cc0f4780 2021-11-20T05:37:42.795 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.793+0000 7f73d2a28700 1 -- 172.21.15.58:0/3293062438 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73cc0f66e0 con 0x7f73cc0f4780 2021-11-20T05:37:42.797 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.794+0000 7f73c8ff9700 1 -- 172.21.15.58:0/3293062438 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f73bc01b060 con 0x7f73cc0f4780 2021-11-20T05:37:42.797 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.794+0000 7f73d2a28700 1 -- 172.21.15.58:0/3293062438 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73cc027a00 con 0x7f73cc0f4780 2021-11-20T05:37:42.797 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.795+0000 7f73c8ff9700 1 --2- 172.21.15.58:0/3293062438 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f73b405c2a0 0x7f73b405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:42.798 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.795+0000 7f73c8ff9700 1 -- 172.21.15.58:0/3293062438 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f73bc0761e0 con 0x7f73cc0f4780 2021-11-20T05:37:42.798 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.795+0000 7f73cb7fe700 1 --2- 172.21.15.58:0/3293062438 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f73b405c2a0 0x7f73b405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:42.798 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.795+0000 7f73cb7fe700 1 --2- 172.21.15.58:0/3293062438 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f73b405c2a0 0x7f73b405e760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f73c0009f50 tx=0x7f73c0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:42.798 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.799+0000 7f73c8ff9700 1 -- 172.21.15.58:0/3293062438 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f73bc048aa0 con 0x7f73cc0f4780 2021-11-20T05:37:42.991 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.995+0000 7f73d2a28700 1 -- 172.21.15.58:0/3293062438 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73cc0586e0 con 0x7f73cc0f4780 2021-11-20T05:37:42.992 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.995+0000 7f73c8ff9700 1 -- 172.21.15.58:0/3293062438 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+2570 (secure 0 0 0) 0x7f73bc04c0c0 con 0x7f73cc0f4780 2021-11-20T05:37:42.992 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:42.993 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:42.395859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24423,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24423":{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":74,"state":"up:replay","state_seq":3,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:42.994 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.997+0000 7f73d2a28700 1 -- 172.21.15.58:0/3293062438 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f73b405c2a0 msgr2=0x7f73b405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:42.994 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.997+0000 7f73d2a28700 1 --2- 172.21.15.58:0/3293062438 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f73b405c2a0 0x7f73b405e760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f73c0009f50 tx=0x7f73c0009b30).stop 2021-11-20T05:37:42.994 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.997+0000 7f73d2a28700 1 -- 172.21.15.58:0/3293062438 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f73cc0f4780 msgr2=0x7f73cc0f7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:42.995 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.997+0000 7f73d2a28700 1 --2- 172.21.15.58:0/3293062438 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f73cc0f4780 0x7f73cc0f7350 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f73bc002700 tx=0x7f73bc009e00).stop 2021-11-20T05:37:42.995 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.997+0000 7f73d2a28700 1 -- 172.21.15.58:0/3293062438 shutdown_connections 2021-11-20T05:37:42.995 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.997+0000 7f73d2a28700 1 --2- 172.21.15.58:0/3293062438 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f73b405c2a0 0x7f73b405e760 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:42.995 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.997+0000 7f73d2a28700 1 --2- 172.21.15.58:0/3293062438 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f73cc0f4780 0x7f73cc0f7350 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:42.996 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.997+0000 7f73d2a28700 1 --2- 172.21.15.58:0/3293062438 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f73cc0f7d90 0x7f73cc0f59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:42.996 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.997+0000 7f73d2a28700 1 -- 172.21.15.58:0/3293062438 >> 172.21.15.58:0/3293062438 conn(0x7f73cc0f04a0 msgr2=0x7f73cc0fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:42.996 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.997+0000 7f73d2a28700 1 -- 172.21.15.58:0/3293062438 shutdown_connections 2021-11-20T05:37:42.996 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:42.997+0000 7f73d2a28700 1 -- 172.21.15.58:0/3293062438 wait complete. 2021-11-20T05:37:42.997 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 74 2021-11-20T05:37:43.005 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 74, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:42.395859+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24423, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24423': {'gid': 24423, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 74, 'state': 'up:replay', 'state_seq': 3, 'addr': '172.21.15.187:6825/2715421480', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 2715421480}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 2715421480}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:43.005 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24423:up:replay 2021-11-20T05:37:43.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:43 smithi187 conmon[36358]: cluster 2021-11-20T05:37:42.288478+0000 mgr.smithi058.zagdlk (mgr.14164) 300 : cluster [DBG] 2021-11-20T05:37:43.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:43 smithi187 conmon[36358]: pgmap v273: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:37:43.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:43 smithi187 conmon[36358]: cluster 2021-11-20T05:37:42.395733+0000 mon.smithi058 (mon.0) 764 : cluster [DBG] mds.? [v2:172.21.15.187:6826/96302868,v1:172.21.15.187:6827/96302868] up:boot 2021-11-20T05:37:43.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:43 smithi187 conmon[36358]: cluster 2021-11-20T05:37:42.395812+0000 mon.smithi058 (mon.0) 765 : 2021-11-20T05:37:43.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:43 smithi187 conmon[36358]: cluster [INF] Standby daemon mds.cephfs.smithi187.wxduhu assigned to filesystem cephfs as rank 0 2021-11-20T05:37:43.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:43 smithi187 conmon[36358]: cluster 2021-11-20T05: 2021-11-20T05:37:43.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:43 smithi187 conmon[36358]: 37:42.396139+0000 mon.smithi058 (mon.0) 766 : cluster 2021-11-20T05:37:43.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:43 smithi187 conmon[36358]: [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby, 1 failed 2021-11-20T05:37:43.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:43 smithi187 conmon[36358]: audit 2021-11-20T05: 2021-11-20T05:37:43.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:43 smithi187 conmon[36358]: 37:42.396373+0000 mon.smithi058 (mon.0) 767 : 2021-11-20T05:37:43.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:43 smithi187 conmon[36358]: audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.fikisa"}]: dispatch 2021-11-20T05:37:43.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:43 smithi187 conmon[36358]: cluster 2021-11-20T05 2021-11-20T05:37:43.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:43 smithi187 conmon[36358]: :37:42.398742+0000 mon.smithi058 (mon.0) 768 : 2021-11-20T05:37:43.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:43 smithi187 conmon[36358]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:43.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:43 smithi187 conmon[36358]: audit 2021-11-20T05 2021-11-20T05:37:43.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:43 smithi187 conmon[36358]: :37:42.996258+0000 mon.smithi058 (mon.0) 769 2021-11-20T05:37:43.698 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:43 smithi187 conmon[36358]: : audit [DBG] from='client.? 172.21.15.58:0/3293062438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:43.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: cluster 2021-11-20T05:37:42.288478+0000 mgr.smithi058.zagdlk (mgr.14164) 300 2021-11-20T05:37:43.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: : cluster [DBG] pgmap v273: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:37:43.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: cluster 2021-11-20T05:37:42.395733+0000 mon.smithi058 (mon.0) 764 : cluster [DBG] mds.? [v2:172.21.15.187:6826/96302868,v1:172.21.15.187:6827/96302868] up:boot 2021-11-20T05:37:43.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: cluster 2021-11-20T 2021-11-20T05:37:43.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: 05:37:42.395812+0000 mon.smithi058 (mon.0) 765 : cluster [INF] Standby daemon mds.cephfs.smithi187.wxduhu assigned to filesystem cephfs as rank 0 2021-11-20T05:37:43.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: cluster 2021-11- 2021-11-20T05:37:43.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: 20T05:37:42.396139+0000 mon.smithi058 (mon.0) 766 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby, 1 failed 2021-11-20T05:37:43.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: 2021-11-20T05:37:43.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: audit 2021-11-20T05:37:43.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: 2021-11- 2021-11-20T05:37:43.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: 20T05:37:42.396373+0000 mon.smithi058 2021-11-20T05:37:43.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: (mon.0) 767 : audit [DBG] 2021-11-20T05:37:43.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.fikisa"}]: dispatch 2021-11-20T05:37:43.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: cluster 2021- 2021-11-20T05:37:43.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: 11-20T05:37:42.398742+0000 mon.smithi058 (mon.0) 768 : 2021-11-20T05:37:43.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:43.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: audit 2021-11-20T05:37:43.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: 2021-11-20T05:37:42.996258+0000 mon.smithi058 (mon.0) 769 : audit [DBG] 2021-11-20T05:37:43.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:43 smithi058 conmon[33537]: from='client.? 172.21.15.58:0/3293062438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:44.007 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:44.141 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.143+0000 7f1824c33700 1 -- 172.21.15.58:0/3617421221 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18200f6820 msgr2=0x7f18200f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:44.142 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.143+0000 7f1824c33700 1 --2- 172.21.15.58:0/3617421221 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18200f6820 0x7f18200f6c80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f1808004660 tx=0x7f1808009b30).stop 2021-11-20T05:37:44.143 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.143+0000 7f1824c33700 1 -- 172.21.15.58:0/3617421221 shutdown_connections 2021-11-20T05:37:44.143 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.143+0000 7f1824c33700 1 --2- 172.21.15.58:0/3617421221 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18200f6820 0x7f18200f6c80 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:44.144 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.143+0000 7f1824c33700 1 --2- 172.21.15.58:0/3617421221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18200f5580 0x7f18200f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:44.144 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.143+0000 7f1824c33700 1 -- 172.21.15.58:0/3617421221 >> 172.21.15.58:0/3617421221 conn(0x7f18200f0ce0 msgr2=0x7f18200f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:44.144 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.143+0000 7f1824c33700 1 -- 172.21.15.58:0/3617421221 shutdown_connections 2021-11-20T05:37:44.144 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.143+0000 7f1824c33700 1 -- 172.21.15.58:0/3617421221 wait complete. 2021-11-20T05:37:44.145 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f1824c33700 1 Processor -- start 2021-11-20T05:37:44.145 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f1824c33700 1 -- start start 2021-11-20T05:37:44.145 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f1824c33700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18200f5580 0x7f1820105dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:44.145 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f1824c33700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18200f6820 0x7f18201062b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:44.146 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f1824c33700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1820106860 con 0x7f18200f5580 2021-11-20T05:37:44.146 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f1824c33700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18201069a0 con 0x7f18200f6820 2021-11-20T05:37:44.146 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f181e59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18200f5580 0x7f1820105dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:44.146 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f181e59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18200f5580 0x7f1820105dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41798/0 (socket says 172.21.15.58:41798) 2021-11-20T05:37:44.147 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f181e59c700 1 -- 172.21.15.58:0/1586894645 learned_addr learned my addr 172.21.15.58:0/1586894645 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:44.147 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f181dd9b700 1 --2- 172.21.15.58:0/1586894645 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18200f6820 0x7f18201062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:44.147 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f181e59c700 1 -- 172.21.15.58:0/1586894645 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18200f6820 msgr2=0x7f18201062b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:44.147 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f181e59c700 1 --2- 172.21.15.58:0/1586894645 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18200f6820 0x7f18201062b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:44.148 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f181e59c700 1 -- 172.21.15.58:0/1586894645 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1808005040 con 0x7f18200f5580 2021-11-20T05:37:44.148 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f181e59c700 1 --2- 172.21.15.58:0/1586894645 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18200f5580 0x7f1820105dc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f1810002700 tx=0x7f1810009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:44.148 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f1816ffd700 1 -- 172.21.15.58:0/1586894645 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1810009320 con 0x7f18200f5580 2021-11-20T05:37:44.148 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f1816ffd700 1 -- 172.21.15.58:0/1586894645 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f181000f040 con 0x7f18200f5580 2021-11-20T05:37:44.149 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f1816ffd700 1 -- 172.21.15.58:0/1586894645 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1810014900 con 0x7f18200f5580 2021-11-20T05:37:44.149 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.144+0000 7f1824c33700 1 -- 172.21.15.58:0/1586894645 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182010b440 con 0x7f18200f5580 2021-11-20T05:37:44.149 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.145+0000 7f1824c33700 1 -- 172.21.15.58:0/1586894645 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182010ba00 con 0x7f18200f5580 2021-11-20T05:37:44.149 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.145+0000 7f1824c33700 1 -- 172.21.15.58:0/1586894645 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1820027a00 con 0x7f18200f5580 2021-11-20T05:37:44.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.146+0000 7f1816ffd700 1 -- 172.21.15.58:0/1586894645 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1810009480 con 0x7f18200f5580 2021-11-20T05:37:44.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.146+0000 7f1816ffd700 1 --2- 172.21.15.58:0/1586894645 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f180c05c2a0 0x7f180c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:44.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.146+0000 7f1816ffd700 1 -- 172.21.15.58:0/1586894645 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f1810076580 con 0x7f18200f5580 2021-11-20T05:37:44.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.146+0000 7f181dd9b700 1 --2- 172.21.15.58:0/1586894645 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f180c05c2a0 0x7f180c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:44.152 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.147+0000 7f181dd9b700 1 --2- 172.21.15.58:0/1586894645 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f180c05c2a0 0x7f180c05e760 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1808005010 tx=0x7f1808009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:44.152 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.150+0000 7f1816ffd700 1 -- 172.21.15.58:0/1586894645 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1810048ec0 con 0x7f18200f5580 2021-11-20T05:37:44.344 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.348+0000 7f1824c33700 1 -- 172.21.15.58:0/1586894645 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18200fdd10 con 0x7f18200f5580 2021-11-20T05:37:44.345 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.348+0000 7f1816ffd700 1 -- 172.21.15.58:0/1586894645 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+2572 (secure 0 0 0) 0x7f181004c4e0 con 0x7f18200f5580 2021-11-20T05:37:44.345 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:44.346 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:43.397782+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24423,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24423":{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":74,"state":"up:resolve","state_seq":12,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:44.347 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.350+0000 7f1824c33700 1 -- 172.21.15.58:0/1586894645 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f180c05c2a0 msgr2=0x7f180c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:44.347 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.350+0000 7f1824c33700 1 --2- 172.21.15.58:0/1586894645 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f180c05c2a0 0x7f180c05e760 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1808005010 tx=0x7f1808009b30).stop 2021-11-20T05:37:44.348 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.350+0000 7f1824c33700 1 -- 172.21.15.58:0/1586894645 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18200f5580 msgr2=0x7f1820105dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:44.348 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.350+0000 7f1824c33700 1 --2- 172.21.15.58:0/1586894645 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18200f5580 0x7f1820105dc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f1810002700 tx=0x7f1810009d80).stop 2021-11-20T05:37:44.348 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.350+0000 7f1824c33700 1 -- 172.21.15.58:0/1586894645 shutdown_connections 2021-11-20T05:37:44.348 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.350+0000 7f1824c33700 1 --2- 172.21.15.58:0/1586894645 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f180c05c2a0 0x7f180c05e760 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:44.349 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.350+0000 7f1824c33700 1 --2- 172.21.15.58:0/1586894645 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18200f5580 0x7f1820105dc0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:44.349 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.350+0000 7f1824c33700 1 --2- 172.21.15.58:0/1586894645 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18200f6820 0x7f18201062b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:44.349 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.350+0000 7f1824c33700 1 -- 172.21.15.58:0/1586894645 >> 172.21.15.58:0/1586894645 conn(0x7f18200f0ce0 msgr2=0x7f18200fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:44.349 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.350+0000 7f1824c33700 1 -- 172.21.15.58:0/1586894645 shutdown_connections 2021-11-20T05:37:44.349 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:44.350+0000 7f1824c33700 1 -- 172.21.15.58:0/1586894645 wait complete. 2021-11-20T05:37:44.350 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 75 2021-11-20T05:37:44.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 75, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:43.397782+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24423, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24423': {'gid': 24423, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 74, 'state': 'up:resolve', 'state_seq': 12, 'addr': '172.21.15.187:6825/2715421480', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 2715421480}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 2715421480}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:44.359 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24423:up:resolve 2021-11-20T05:37:44.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:44 smithi187 conmon[36358]: cluster 2021-11-20T05:37:43.400680+0000 mon.smithi058 (mon.0) 770 : cluster [DBG] mds.? [v2:172.21.15.187:6824/2715421480,v1:172.21.15.187:6825/2715421480] up:resolve 2021-11-20T05:37:44.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:44 smithi187 conmon[36358]: cluster 2021-11-20T05:37:43.400770+0000 mon.smithi058 (mon.0) 771 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:44.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:44 smithi187 conmon[36358]: audit 2021-11-20T05:37:44.349476+0000 mon.smithi058 (mon.0) 772 : audit [DBG] from='client.? 172.21.15.58:0/1586894645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:44.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:44 smithi058 conmon[33537]: cluster 2021-11-20T05:37:43.400680+0000 mon.smithi058 (mon.0) 770 : cluster 2021-11-20T05:37:44.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:44 smithi058 conmon[33537]: [DBG] mds.? [v2:172.21.15.187:6824/2715421480,v1:172.21.15.187:6825/2715421480] up:resolve 2021-11-20T05:37:44.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:44 smithi058 conmon[33537]: cluster 2021-11-20T05:37:43.400770+0000 mon.smithi058 (mon.0) 771 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:44.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:44 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:37:44.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:44 smithi058 conmon[33537]: -20T05:37:44.349476+0000 mon.smithi058 (mon.0) 772 : audit [DBG] from='client.? 172.21.15.58:0/1586894645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:45.359 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:45.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:45 smithi058 conmon[33537]: cluster 2021-11-20T05:37:44.288900+0000 mgr.smithi058.zagdlk (mgr.14164) 301 : cluster [DBG] pgmap v275: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:37:45.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:45 smithi058 conmon[33537]: cluster 2021-11 2021-11-20T05:37:45.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:45 smithi058 conmon[33537]: -20T05:37:44.402462+0000 mon.smithi058 (mon.0) 773 : cluster [DBG] mds.? [v2:172.21.15.187:6824/2715421480,v1:172.21.15.187:6825/2715421480] up:reconnect 2021-11-20T05:37:45.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:45 smithi058 conmon[33537]: cluster 2021-11- 2021-11-20T05:37:45.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:45 smithi058 conmon[33537]: 20T05:37:44.402529+0000 mon.smithi058 (mon.0) 774 : cluster 2021-11-20T05:37:45.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:45 smithi058 conmon[33537]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:45.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:45 smithi058 conmon[33537]: cluster 2021-11-20T05:37 2021-11-20T05:37:45.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:45 smithi058 conmon[33537]: :44.422439+0000 mds.cephfs.smithi187.wxduhu (mds.0) 1 2021-11-20T05:37:45.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:45 smithi058 conmon[33537]: : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0209997 2021-11-20T05:37:45.498 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.495+0000 7fb122279700 1 -- 172.21.15.58:0/2542569828 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb11c0f54d0 msgr2=0x7fb11c0f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:45.499 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.495+0000 7fb122279700 1 --2- 172.21.15.58:0/2542569828 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb11c0f54d0 0x7fb11c0f58f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fb118004660 tx=0x7fb118009b30).stop 2021-11-20T05:37:45.499 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.495+0000 7fb122279700 1 -- 172.21.15.58:0/2542569828 shutdown_connections 2021-11-20T05:37:45.500 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.495+0000 7fb122279700 1 --2- 172.21.15.58:0/2542569828 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb11c0f6770 0x7fb11c0f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:45.500 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.495+0000 7fb122279700 1 --2- 172.21.15.58:0/2542569828 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb11c0f54d0 0x7fb11c0f58f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:45.500 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.495+0000 7fb122279700 1 -- 172.21.15.58:0/2542569828 >> 172.21.15.58:0/2542569828 conn(0x7fb11c0f0c30 msgr2=0x7fb11c0f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:45.501 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.495+0000 7fb122279700 1 -- 172.21.15.58:0/2542569828 shutdown_connections 2021-11-20T05:37:45.502 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.495+0000 7fb122279700 1 -- 172.21.15.58:0/2542569828 wait complete. 2021-11-20T05:37:45.502 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.496+0000 7fb122279700 1 Processor -- start 2021-11-20T05:37:45.502 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.496+0000 7fb122279700 1 -- start start 2021-11-20T05:37:45.503 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.496+0000 7fb122279700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb11c0f54d0 0x7fb11c106dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:45.503 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.496+0000 7fb122279700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb11c0f6770 0x7fb11c1072b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:45.503 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.496+0000 7fb122279700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c1078b0 con 0x7fb11c0f6770 2021-11-20T05:37:45.504 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.496+0000 7fb122279700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c1080f0 con 0x7fb11c0f54d0 2021-11-20T05:37:45.504 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.496+0000 7fb120a76700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb11c0f6770 0x7fb11c1072b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:45.504 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.496+0000 7fb120a76700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb11c0f6770 0x7fb11c1072b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41814/0 (socket says 172.21.15.58:41814) 2021-11-20T05:37:45.505 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.496+0000 7fb120a76700 1 -- 172.21.15.58:0/1621844910 learned_addr learned my addr 172.21.15.58:0/1621844910 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:45.505 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.496+0000 7fb121277700 1 --2- 172.21.15.58:0/1621844910 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb11c0f54d0 0x7fb11c106dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:45.506 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.497+0000 7fb120a76700 1 -- 172.21.15.58:0/1621844910 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb11c0f54d0 msgr2=0x7fb11c106dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:45.506 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.497+0000 7fb120a76700 1 --2- 172.21.15.58:0/1621844910 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb11c0f54d0 0x7fb11c106dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:45.506 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.497+0000 7fb120a76700 1 -- 172.21.15.58:0/1621844910 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb118005040 con 0x7fb11c0f6770 2021-11-20T05:37:45.506 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.497+0000 7fb121277700 1 --2- 172.21.15.58:0/1621844910 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb11c0f54d0 0x7fb11c106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:37:45.507 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.497+0000 7fb120a76700 1 --2- 172.21.15.58:0/1621844910 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb11c0f6770 0x7fb11c1072b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fb10c00c510 tx=0x7fb10c007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:45.507 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.497+0000 7fb111ffb700 1 -- 172.21.15.58:0/1621844910 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb10c00f650 con 0x7fb11c0f6770 2021-11-20T05:37:45.507 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.497+0000 7fb111ffb700 1 -- 172.21.15.58:0/1621844910 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb10c00fc60 con 0x7fb11c0f6770 2021-11-20T05:37:45.507 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.497+0000 7fb122279700 1 -- 172.21.15.58:0/1621844910 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb11c1083d0 con 0x7fb11c0f6770 2021-11-20T05:37:45.508 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.497+0000 7fb111ffb700 1 -- 172.21.15.58:0/1621844910 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb10c012cb0 con 0x7fb11c0f6770 2021-11-20T05:37:45.508 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.497+0000 7fb122279700 1 -- 172.21.15.58:0/1621844910 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb11c108910 con 0x7fb11c0f6770 2021-11-20T05:37:45.508 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.498+0000 7fb122279700 1 -- 172.21.15.58:0/1621844910 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb11c0f48c0 con 0x7fb11c0f6770 2021-11-20T05:37:45.508 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.498+0000 7fb111ffb700 1 -- 172.21.15.58:0/1621844910 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb10c010070 con 0x7fb11c0f6770 2021-11-20T05:37:45.509 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.499+0000 7fb111ffb700 1 --2- 172.21.15.58:0/1621844910 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb10805c300 0x7fb10805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:45.509 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.499+0000 7fb111ffb700 1 -- 172.21.15.58:0/1621844910 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fb10c076070 con 0x7fb11c0f6770 2021-11-20T05:37:45.509 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.499+0000 7fb121277700 1 --2- 172.21.15.58:0/1621844910 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb10805c300 0x7fb10805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:45.509 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.499+0000 7fb121277700 1 --2- 172.21.15.58:0/1621844910 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb10805c300 0x7fb10805e7c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fb11c107b80 tx=0x7fb118009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:45.510 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.503+0000 7fb111ffb700 1 -- 172.21.15.58:0/1621844910 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb10c0452b0 con 0x7fb11c0f6770 2021-11-20T05:37:45.652 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:45 smithi187 conmon[36358]: cluster 2021-11-20T05:37:44.288900+0000 mgr.smithi058.zagdlk (mgr.14164) 301 2021-11-20T05:37:45.653 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:45 smithi187 conmon[36358]: : cluster [DBG] pgmap v275: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:37:45.653 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:45 smithi187 conmon[36358]: cluster 2021-11-20T05:37:44.402462+0000 mon.smithi058 (mon.0) 773 : cluster [DBG] mds.? [v2:172.21.15.187:6824/2715421480,v1:172.21.15.187:6825/2715421480] up:reconnect 2021-11-20T05:37:45.653 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:45 smithi187 conmon[36358]: cluster 2021-11-20T05:37:44.402529+0000 mon.smithi058 (mon.0) 774 2021-11-20T05:37:45.653 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:45 smithi187 conmon[36358]: : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:45.654 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:45 smithi187 conmon[36358]: cluster 2021-11-20T05: 2021-11-20T05:37:45.654 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:45 smithi187 conmon[36358]: 37:44.422439+0000 mds.cephfs.smithi187.wxduhu (mds.0) 1 2021-11-20T05:37:45.654 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:45 smithi187 conmon[36358]: : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0209997 2021-11-20T05:37:45.702 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.706+0000 7fb122279700 1 -- 172.21.15.58:0/1621844910 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb11c108f50 con 0x7fb11c0f6770 2021-11-20T05:37:45.703 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.707+0000 7fb111ffb700 1 -- 172.21.15.58:0/1621844910 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2571 (secure 0 0 0) 0x7fb10c04c360 con 0x7fb11c0f6770 2021-11-20T05:37:45.703 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:45.704 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:45.403981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24423,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24423":{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:45.706 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.708+0000 7fb122279700 1 -- 172.21.15.58:0/1621844910 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb10805c300 msgr2=0x7fb10805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:45.706 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.708+0000 7fb122279700 1 --2- 172.21.15.58:0/1621844910 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb10805c300 0x7fb10805e7c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fb11c107b80 tx=0x7fb118009b30).stop 2021-11-20T05:37:45.707 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.708+0000 7fb122279700 1 -- 172.21.15.58:0/1621844910 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb11c0f6770 msgr2=0x7fb11c1072b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:45.707 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.708+0000 7fb122279700 1 --2- 172.21.15.58:0/1621844910 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb11c0f6770 0x7fb11c1072b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fb10c00c510 tx=0x7fb10c007e00).stop 2021-11-20T05:37:45.707 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.708+0000 7fb122279700 1 -- 172.21.15.58:0/1621844910 shutdown_connections 2021-11-20T05:37:45.707 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.708+0000 7fb122279700 1 --2- 172.21.15.58:0/1621844910 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb10805c300 0x7fb10805e7c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:45.708 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.708+0000 7fb122279700 1 --2- 172.21.15.58:0/1621844910 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb11c0f54d0 0x7fb11c106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:45.708 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.708+0000 7fb122279700 1 --2- 172.21.15.58:0/1621844910 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb11c0f6770 0x7fb11c1072b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:45.708 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.708+0000 7fb122279700 1 -- 172.21.15.58:0/1621844910 >> 172.21.15.58:0/1621844910 conn(0x7fb11c0f0c30 msgr2=0x7fb11c05f780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:45.708 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.708+0000 7fb122279700 1 -- 172.21.15.58:0/1621844910 shutdown_connections 2021-11-20T05:37:45.708 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:45.708+0000 7fb122279700 1 -- 172.21.15.58:0/1621844910 wait complete. 2021-11-20T05:37:45.709 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 77 2021-11-20T05:37:45.717 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:45.403981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24423, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24423': {'gid': 24423, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 14, 'addr': '172.21.15.187:6825/2715421480', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 2715421480}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 2715421480}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:45.718 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24423:up:rejoin 2021-11-20T05:37:46.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:46 smithi187 conmon[36358]: cluster 2021-11-20T05:37:45.407628+0000 mon.smithi058 (mon.0) 775 : cluster [DBG] 2021-11-20T05:37:46.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:46 smithi187 conmon[36358]: mds.? [v2:172.21.15.187:6824/2715421480,v1:172.21.15.187:6825/2715421480] up:rejoin 2021-11-20T05:37:46.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:46 smithi187 conmon[36358]: cluster 2021-11-20T05:37:45.407706+0000 mon.smithi058 (mon.0) 776 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:46.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:46 smithi187 conmon[36358]: audit 2021-11-20T05:37:45.707579+0000 mon.smithi058 (mon.0) 777 : audit [DBG] from='client.? 172.21.15.58:0/1621844910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:46.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:46 smithi058 conmon[33537]: cluster 2021-11-20T05:37:45.407628+0000 mon.smithi058 (mon.0) 775 : cluster [DBG] mds.? [v2:172.21.15.187:6824/2715421480,v1:172.21.15.187:6825/2715421480] up:rejoin 2021-11-20T05:37:46.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:46 smithi058 conmon[33537]: cluster 2021-11-20T05:37:45.407706+0000 mon.smithi058 (mon.0) 776 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:46.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:46 smithi058 conmon[33537]: audit 2021-11-20T05:37:45.707579+0000 2021-11-20T05:37:46.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:46 smithi058 conmon[33537]: mon.smithi058 (mon.0) 777 : audit [DBG] from='client.? 172.21.15.58:0/1621844910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:46.718 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:46.856 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.855+0000 7fd3371fa700 1 -- 172.21.15.58:0/192178326 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd3300f7870 msgr2=0x7fd3300f7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:46.856 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.855+0000 7fd3371fa700 1 --2- 172.21.15.58:0/192178326 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd3300f7870 0x7fd3300f7cd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fd320004660 tx=0x7fd320009b30).stop 2021-11-20T05:37:46.856 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.855+0000 7fd3371fa700 1 -- 172.21.15.58:0/192178326 shutdown_connections 2021-11-20T05:37:46.857 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.855+0000 7fd3371fa700 1 --2- 172.21.15.58:0/192178326 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd3300f7870 0x7fd3300f7cd0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:46.857 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.855+0000 7fd3371fa700 1 --2- 172.21.15.58:0/192178326 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3300f65d0 0x7fd3300f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:46.857 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.855+0000 7fd3371fa700 1 -- 172.21.15.58:0/192178326 >> 172.21.15.58:0/192178326 conn(0x7fd3300f1d10 msgr2=0x7fd3300f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:46.858 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.855+0000 7fd3371fa700 1 -- 172.21.15.58:0/192178326 shutdown_connections 2021-11-20T05:37:46.858 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.855+0000 7fd3371fa700 1 -- 172.21.15.58:0/192178326 wait complete. 2021-11-20T05:37:46.858 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.855+0000 7fd3371fa700 1 Processor -- start 2021-11-20T05:37:46.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.855+0000 7fd3371fa700 1 -- start start 2021-11-20T05:37:46.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.856+0000 7fd3371fa700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd3300f65d0 0x7fd330106d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:46.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.856+0000 7fd3371fa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3300f7870 0x7fd330107280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:46.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.856+0000 7fd3371fa700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd330107880 con 0x7fd3300f65d0 2021-11-20T05:37:46.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.856+0000 7fd3371fa700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd330108090 con 0x7fd3300f7870 2021-11-20T05:37:46.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.856+0000 7fd3361f8700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd3300f65d0 0x7fd330106d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:46.862 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.856+0000 7fd3361f8700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd3300f65d0 0x7fd330106d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41834/0 (socket says 172.21.15.58:41834) 2021-11-20T05:37:46.862 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.856+0000 7fd3361f8700 1 -- 172.21.15.58:0/2963696903 learned_addr learned my addr 172.21.15.58:0/2963696903 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:46.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.856+0000 7fd3361f8700 1 -- 172.21.15.58:0/2963696903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3300f7870 msgr2=0x7fd330107280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:46.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.856+0000 7fd3361f8700 1 --2- 172.21.15.58:0/2963696903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3300f7870 0x7fd330107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:37:46.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.856+0000 7fd3361f8700 1 -- 172.21.15.58:0/2963696903 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd320005040 con 0x7fd3300f65d0 2021-11-20T05:37:46.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.856+0000 7fd3361f8700 1 --2- 172.21.15.58:0/2963696903 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd3300f65d0 0x7fd330106d90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fd32c002700 tx=0x7fd32c009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:46.864 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.856+0000 7fd326ffd700 1 -- 172.21.15.58:0/2963696903 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd32c009320 con 0x7fd3300f65d0 2021-11-20T05:37:46.864 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.856+0000 7fd326ffd700 1 -- 172.21.15.58:0/2963696903 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd32c00f040 con 0x7fd3300f65d0 2021-11-20T05:37:46.864 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.856+0000 7fd3371fa700 1 -- 172.21.15.58:0/2963696903 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd330108310 con 0x7fd3300f65d0 2021-11-20T05:37:46.865 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.856+0000 7fd326ffd700 1 -- 172.21.15.58:0/2963696903 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd32c014900 con 0x7fd3300f65d0 2021-11-20T05:37:46.865 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.856+0000 7fd3371fa700 1 -- 172.21.15.58:0/2963696903 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3301088d0 con 0x7fd3300f65d0 2021-11-20T05:37:46.865 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.857+0000 7fd3371fa700 1 -- 172.21.15.58:0/2963696903 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3300f59c0 con 0x7fd3300f65d0 2021-11-20T05:37:46.865 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.858+0000 7fd326ffd700 1 -- 172.21.15.58:0/2963696903 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd32c009480 con 0x7fd3300f65d0 2021-11-20T05:37:46.866 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.858+0000 7fd326ffd700 1 --2- 172.21.15.58:0/2963696903 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd31c05c300 0x7fd31c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:46.866 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.858+0000 7fd326ffd700 1 -- 172.21.15.58:0/2963696903 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fd32c076610 con 0x7fd3300f65d0 2021-11-20T05:37:46.866 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.858+0000 7fd3359f7700 1 --2- 172.21.15.58:0/2963696903 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd31c05c300 0x7fd31c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:46.866 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.859+0000 7fd3359f7700 1 --2- 172.21.15.58:0/2963696903 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd31c05c300 0x7fd31c05e7c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fd320009f50 tx=0x7fd320009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:46.867 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:46.862+0000 7fd326ffd700 1 -- 172.21.15.58:0/2963696903 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd32c048ed0 con 0x7fd3300f65d0 2021-11-20T05:37:47.054 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:47.058+0000 7fd3371fa700 1 -- 172.21.15.58:0/2963696903 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd330045ad0 con 0x7fd3300f65d0 2021-11-20T05:37:47.055 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:47.058+0000 7fd326ffd700 1 -- 172.21.15.58:0/2963696903 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2570 (secure 0 0 0) 0x7fd32c04c4f0 con 0x7fd3300f65d0 2021-11-20T05:37:47.055 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:47.056 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:45.403981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24423,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24423":{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:47.057 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:47.060+0000 7fd3371fa700 1 -- 172.21.15.58:0/2963696903 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd31c05c300 msgr2=0x7fd31c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:47.058 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:47.060+0000 7fd3371fa700 1 --2- 172.21.15.58:0/2963696903 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd31c05c300 0x7fd31c05e7c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fd320009f50 tx=0x7fd320009b30).stop 2021-11-20T05:37:47.058 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:47.060+0000 7fd3371fa700 1 -- 172.21.15.58:0/2963696903 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd3300f65d0 msgr2=0x7fd330106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:47.059 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:47.060+0000 7fd3371fa700 1 --2- 172.21.15.58:0/2963696903 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd3300f65d0 0x7fd330106d90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fd32c002700 tx=0x7fd32c009d80).stop 2021-11-20T05:37:47.059 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:47.060+0000 7fd3371fa700 1 -- 172.21.15.58:0/2963696903 shutdown_connections 2021-11-20T05:37:47.059 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:47.060+0000 7fd3371fa700 1 --2- 172.21.15.58:0/2963696903 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd31c05c300 0x7fd31c05e7c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:47.060 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:47.060+0000 7fd3371fa700 1 --2- 172.21.15.58:0/2963696903 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd3300f65d0 0x7fd330106d90 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:47.060 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:47.060+0000 7fd3371fa700 1 --2- 172.21.15.58:0/2963696903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3300f7870 0x7fd330107280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:37:47.060 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:47.060+0000 7fd3371fa700 1 -- 172.21.15.58:0/2963696903 >> 172.21.15.58:0/2963696903 conn(0x7fd3300f1d10 msgr2=0x7fd3300fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:47.060 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:47.060+0000 7fd3371fa700 1 -- 172.21.15.58:0/2963696903 shutdown_connections 2021-11-20T05:37:47.061 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:47.060+0000 7fd3371fa700 1 -- 172.21.15.58:0/2963696903 wait complete. 2021-11-20T05:37:47.062 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 78 2021-11-20T05:37:47.069 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:45.403981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24423, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24423': {'gid': 24423, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 14, 'addr': '172.21.15.187:6825/2715421480', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 2715421480}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 2715421480}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:47.069 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24423:up:rejoin 2021-11-20T05:37:47.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:47 smithi187 conmon[36358]: cluster 2021-11-20T05:37:46.289421+0000 2021-11-20T05:37:47.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:47 smithi187 conmon[36358]: mgr.smithi058.zagdlk (mgr.14164) 302 : cluster [DBG] pgmap v276: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:37:47.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:47 smithi187 conmon[36358]: cluster 2021-11-20T05:37:46.423028+0000 mon.smithi058 ( 2021-11-20T05:37:47.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:47 smithi187 conmon[36358]: mon.0) 778 : cluster [DBG] mds.? [v2:172.21.15.187:6826/96302868,v1:172.21.15.187:6827/96302868] up:standby 2021-11-20T05:37:47.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:47 smithi187 conmon[36358]: cluster 2021-11-20T05:37:46.423119+0000 mon.smithi058 ( 2021-11-20T05:37:47.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:47 smithi187 conmon[36358]: mon.0) 779 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:47.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:47 smithi187 conmon[36358]: audit 2021-11-20T05:37:47.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:47 smithi187 conmon[36358]: 2021-11-20T05:37:47.059298+0000 mon.smithi058 (mon 2021-11-20T05:37:47.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:47 smithi187 conmon[36358]: .0) 780 : audit [DBG] from='client.? 172.21.15.58:0/2963696903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:47.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:47 smithi058 conmon[33537]: cluster 2021-11-20T05:37:46.289421+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:37:47.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:47 smithi058 conmon[33537]: 302 : cluster [DBG] pgmap v276: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:37:47.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:47 smithi058 conmon[33537]: cluster 2021-11-20T05:37:46.423028+0000 mon.smithi058 (mon.0) 778 : cluster [DBG] mds.? [v2:172.21.15.187:6826/96302868,v1:172.21.15.187:6827/96302868] up:standby 2021-11-20T05:37:47.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:47 smithi058 conmon[33537]: cluster 2021-11-20T 2021-11-20T05:37:47.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:47 smithi058 conmon[33537]: 05:37:46.423119+0000 2021-11-20T05:37:47.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:47 smithi058 conmon[33537]: mon.smithi058 (mon.0) 779 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:47.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:47 smithi058 conmon[33537]: audit 2021-11- 2021-11-20T05:37:47.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:47 smithi058 conmon[33537]: 20T05:37:47.059298+0000 mon.smithi058 (mon 2021-11-20T05:37:47.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:47 smithi058 conmon[33537]: .0) 780 : audit [DBG] from='client.? 172.21.15.58:0/2963696903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:48.071 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:48.211 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.209+0000 7fe9d7f96700 1 -- 172.21.15.58:0/536272536 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe9d00f5580 msgr2=0x7fe9d00f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:48.212 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.209+0000 7fe9d7f96700 1 --2- 172.21.15.58:0/536272536 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe9d00f5580 0x7fe9d00f59a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fe9c8004660 tx=0x7fe9c8009b30).stop 2021-11-20T05:37:48.212 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.209+0000 7fe9d7f96700 1 -- 172.21.15.58:0/536272536 shutdown_connections 2021-11-20T05:37:48.212 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.209+0000 7fe9d7f96700 1 --2- 172.21.15.58:0/536272536 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9d00f6820 0x7fe9d00f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:48.213 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.209+0000 7fe9d7f96700 1 --2- 172.21.15.58:0/536272536 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe9d00f5580 0x7fe9d00f59a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:48.213 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.209+0000 7fe9d7f96700 1 -- 172.21.15.58:0/536272536 >> 172.21.15.58:0/536272536 conn(0x7fe9d00f0ce0 msgr2=0x7fe9d00f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:48.213 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.209+0000 7fe9d7f96700 1 -- 172.21.15.58:0/536272536 shutdown_connections 2021-11-20T05:37:48.214 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.209+0000 7fe9d7f96700 1 -- 172.21.15.58:0/536272536 wait complete. 2021-11-20T05:37:48.214 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.210+0000 7fe9d7f96700 1 Processor -- start 2021-11-20T05:37:48.214 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.210+0000 7fe9d7f96700 1 -- start start 2021-11-20T05:37:48.215 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.210+0000 7fe9d7f96700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9d00f5580 0x7fe9d0060530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:48.215 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.210+0000 7fe9d7f96700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe9d00f6820 0x7fe9d010bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:48.216 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.210+0000 7fe9d7f96700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d010c150 con 0x7fe9d00f6820 2021-11-20T05:37:48.216 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.210+0000 7fe9d7f96700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d010c290 con 0x7fe9d00f5580 2021-11-20T05:37:48.216 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.210+0000 7fe9d5531700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe9d00f6820 0x7fe9d010bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:48.217 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.210+0000 7fe9d5531700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe9d00f6820 0x7fe9d010bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41848/0 (socket says 172.21.15.58:41848) 2021-11-20T05:37:48.217 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.210+0000 7fe9d5531700 1 -- 172.21.15.58:0/1177206853 learned_addr learned my addr 172.21.15.58:0/1177206853 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:48.218 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.210+0000 7fe9d5d32700 1 --2- 172.21.15.58:0/1177206853 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9d00f5580 0x7fe9d0060530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:48.218 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.211+0000 7fe9d5531700 1 -- 172.21.15.58:0/1177206853 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9d00f5580 msgr2=0x7fe9d0060530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:48.218 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.211+0000 7fe9d5531700 1 --2- 172.21.15.58:0/1177206853 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9d00f5580 0x7fe9d0060530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:48.218 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.211+0000 7fe9d5531700 1 -- 172.21.15.58:0/1177206853 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c8005040 con 0x7fe9d00f6820 2021-11-20T05:37:48.219 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.211+0000 7fe9d5531700 1 --2- 172.21.15.58:0/1177206853 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe9d00f6820 0x7fe9d010bc60 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fe9c40026d0 tx=0x7fe9c4009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:48.219 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.211+0000 7fe9ce7fc700 1 -- 172.21.15.58:0/1177206853 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9c40079c0 con 0x7fe9d00f6820 2021-11-20T05:37:48.219 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.211+0000 7fe9ce7fc700 1 -- 172.21.15.58:0/1177206853 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9c400f040 con 0x7fe9d00f6820 2021-11-20T05:37:48.219 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.211+0000 7fe9ce7fc700 1 -- 172.21.15.58:0/1177206853 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9c40148b0 con 0x7fe9d00f6820 2021-11-20T05:37:48.220 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.211+0000 7fe9d7f96700 1 -- 172.21.15.58:0/1177206853 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d010c520 con 0x7fe9d00f6820 2021-11-20T05:37:48.220 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.211+0000 7fe9d7f96700 1 -- 172.21.15.58:0/1177206853 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d010ca90 con 0x7fe9d00f6820 2021-11-20T05:37:48.220 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.212+0000 7fe9d7f96700 1 -- 172.21.15.58:0/1177206853 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d0027a00 con 0x7fe9d00f6820 2021-11-20T05:37:48.221 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.212+0000 7fe9ce7fc700 1 -- 172.21.15.58:0/1177206853 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe9c4007b20 con 0x7fe9d00f6820 2021-11-20T05:37:48.221 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.213+0000 7fe9ce7fc700 1 --2- 172.21.15.58:0/1177206853 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe9b805c2a0 0x7fe9b805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:48.222 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.213+0000 7fe9ce7fc700 1 -- 172.21.15.58:0/1177206853 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fe9c4076240 con 0x7fe9d00f6820 2021-11-20T05:37:48.222 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.213+0000 7fe9d5d32700 1 --2- 172.21.15.58:0/1177206853 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe9b805c2a0 0x7fe9b805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:48.222 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.213+0000 7fe9d5d32700 1 --2- 172.21.15.58:0/1177206853 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe9b805c2a0 0x7fe9b805e760 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fe9c8009f50 tx=0x7fe9c8009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:48.223 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.217+0000 7fe9ce7fc700 1 -- 172.21.15.58:0/1177206853 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9c4048b00 con 0x7fe9d00f6820 2021-11-20T05:37:48.411 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.415+0000 7fe9d7f96700 1 -- 172.21.15.58:0/1177206853 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9d00fdd10 con 0x7fe9d00f6820 2021-11-20T05:37:48.412 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.416+0000 7fe9ce7fc700 1 -- 172.21.15.58:0/1177206853 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2570 (secure 0 0 0) 0x7fe9c404c120 con 0x7fe9d00f6820 2021-11-20T05:37:48.413 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:48.413 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:45.403981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24423,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24423":{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:48.414 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.417+0000 7fe9d7f96700 1 -- 172.21.15.58:0/1177206853 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe9b805c2a0 msgr2=0x7fe9b805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:48.415 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.417+0000 7fe9d7f96700 1 --2- 172.21.15.58:0/1177206853 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe9b805c2a0 0x7fe9b805e760 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fe9c8009f50 tx=0x7fe9c8009b30).stop 2021-11-20T05:37:48.415 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.417+0000 7fe9d7f96700 1 -- 172.21.15.58:0/1177206853 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe9d00f6820 msgr2=0x7fe9d010bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:48.415 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.417+0000 7fe9d7f96700 1 --2- 172.21.15.58:0/1177206853 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe9d00f6820 0x7fe9d010bc60 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fe9c40026d0 tx=0x7fe9c4009de0).stop 2021-11-20T05:37:48.415 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.417+0000 7fe9d7f96700 1 -- 172.21.15.58:0/1177206853 shutdown_connections 2021-11-20T05:37:48.416 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.417+0000 7fe9d7f96700 1 --2- 172.21.15.58:0/1177206853 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe9b805c2a0 0x7fe9b805e760 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:48.416 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.418+0000 7fe9d7f96700 1 --2- 172.21.15.58:0/1177206853 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9d00f5580 0x7fe9d0060530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:48.416 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.418+0000 7fe9d7f96700 1 --2- 172.21.15.58:0/1177206853 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe9d00f6820 0x7fe9d010bc60 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:48.416 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.418+0000 7fe9d7f96700 1 -- 172.21.15.58:0/1177206853 >> 172.21.15.58:0/1177206853 conn(0x7fe9d00f0ce0 msgr2=0x7fe9d00fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:48.416 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.418+0000 7fe9d7f96700 1 -- 172.21.15.58:0/1177206853 shutdown_connections 2021-11-20T05:37:48.418 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:48.418+0000 7fe9d7f96700 1 -- 172.21.15.58:0/1177206853 wait complete. 2021-11-20T05:37:48.418 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 78 2021-11-20T05:37:48.426 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:45.403981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24423, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24423': {'gid': 24423, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 14, 'addr': '172.21.15.187:6825/2715421480', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 2715421480}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 2715421480}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:48.427 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24423:up:rejoin 2021-11-20T05:37:48.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:48 smithi187 conmon[36358]: audit 2021-11-20T05:37:48.416987+0000 mon.smithi058 (mon.0 2021-11-20T05:37:48.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:48 smithi187 conmon[36358]: ) 781 : audit [DBG] from='client.? 172.21.15.58:0/1177206853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:48.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:48 smithi058 conmon[33537]: audit 2021-11-20T05:37:48.416987+0000 2021-11-20T05:37:48.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:48 smithi058 conmon[33537]: mon.smithi058 (mon.0) 781 : audit [DBG] from='client.? 172.21.15.58:0/1177206853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:49.428 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:49.562 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.561+0000 7fdfb6ac2700 1 -- 172.21.15.58:0/2551317690 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdfb00f7840 msgr2=0x7fdfb00f7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:49.562 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.561+0000 7fdfb6ac2700 1 --2- 172.21.15.58:0/2551317690 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdfb00f7840 0x7fdfb00f7ca0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fdfa0004660 tx=0x7fdfa0009b30).stop 2021-11-20T05:37:49.562 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.562+0000 7fdfb6ac2700 1 -- 172.21.15.58:0/2551317690 shutdown_connections 2021-11-20T05:37:49.563 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.562+0000 7fdfb6ac2700 1 --2- 172.21.15.58:0/2551317690 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdfb00f7840 0x7fdfb00f7ca0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:49.563 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.562+0000 7fdfb6ac2700 1 --2- 172.21.15.58:0/2551317690 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb00f65a0 0x7fdfb00f69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:49.563 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.562+0000 7fdfb6ac2700 1 -- 172.21.15.58:0/2551317690 >> 172.21.15.58:0/2551317690 conn(0x7fdfb00f1ce0 msgr2=0x7fdfb00f4120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:49.563 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.562+0000 7fdfb6ac2700 1 -- 172.21.15.58:0/2551317690 shutdown_connections 2021-11-20T05:37:49.563 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.562+0000 7fdfb6ac2700 1 -- 172.21.15.58:0/2551317690 wait complete. 2021-11-20T05:37:49.564 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.562+0000 7fdfb6ac2700 1 Processor -- start 2021-11-20T05:37:49.564 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.562+0000 7fdfb6ac2700 1 -- start start 2021-11-20T05:37:49.565 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.562+0000 7fdfb6ac2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb00f65a0 0x7fdfb0106d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:49.565 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.562+0000 7fdfb6ac2700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdfb00f7840 0x7fdfb0107270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:49.565 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.562+0000 7fdfb6ac2700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb0107870 con 0x7fdfb00f7840 2021-11-20T05:37:49.566 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.562+0000 7fdfb6ac2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb01080b0 con 0x7fdfb00f65a0 2021-11-20T05:37:49.566 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.562+0000 7fdfb52bf700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdfb00f7840 0x7fdfb0107270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:49.566 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.563+0000 7fdfb52bf700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdfb00f7840 0x7fdfb0107270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41862/0 (socket says 172.21.15.58:41862) 2021-11-20T05:37:49.566 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.563+0000 7fdfb52bf700 1 -- 172.21.15.58:0/4000073478 learned_addr learned my addr 172.21.15.58:0/4000073478 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:49.566 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.563+0000 7fdfb52bf700 1 -- 172.21.15.58:0/4000073478 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb00f65a0 msgr2=0x7fdfb0106d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:49.567 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.563+0000 7fdfb52bf700 1 --2- 172.21.15.58:0/4000073478 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb00f65a0 0x7fdfb0106d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:37:49.567 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.563+0000 7fdfb52bf700 1 -- 172.21.15.58:0/4000073478 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa0005040 con 0x7fdfb00f7840 2021-11-20T05:37:49.567 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.563+0000 7fdfb52bf700 1 --2- 172.21.15.58:0/4000073478 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdfb00f7840 0x7fdfb0107270 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fdfa0009f50 tx=0x7fdfa0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:49.567 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.563+0000 7fdfa67fc700 1 -- 172.21.15.58:0/4000073478 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfa0018070 con 0x7fdfb00f7840 2021-11-20T05:37:49.568 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.563+0000 7fdfa67fc700 1 -- 172.21.15.58:0/4000073478 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfa0013410 con 0x7fdfb00f7840 2021-11-20T05:37:49.568 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.563+0000 7fdfb6ac2700 1 -- 172.21.15.58:0/4000073478 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb0108330 con 0x7fdfb00f7840 2021-11-20T05:37:49.568 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.563+0000 7fdfa67fc700 1 -- 172.21.15.58:0/4000073478 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfa001c3c0 con 0x7fdfb00f7840 2021-11-20T05:37:49.568 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.563+0000 7fdfb6ac2700 1 -- 172.21.15.58:0/4000073478 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb0108810 con 0x7fdfb00f7840 2021-11-20T05:37:49.569 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.564+0000 7fdfb6ac2700 1 -- 172.21.15.58:0/4000073478 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb00f5990 con 0x7fdfb00f7840 2021-11-20T05:37:49.569 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.564+0000 7fdfa67fc700 1 -- 172.21.15.58:0/4000073478 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fdfa001d420 con 0x7fdfb00f7840 2021-11-20T05:37:49.569 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.565+0000 7fdfa67fc700 1 --2- 172.21.15.58:0/4000073478 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdf9c05c300 0x7fdf9c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:49.569 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.565+0000 7fdfa67fc700 1 -- 172.21.15.58:0/4000073478 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fdfa0078080 con 0x7fdfb00f7840 2021-11-20T05:37:49.569 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.565+0000 7fdfb5ac0700 1 --2- 172.21.15.58:0/4000073478 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdf9c05c300 0x7fdf9c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:49.570 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.565+0000 7fdfb5ac0700 1 --2- 172.21.15.58:0/4000073478 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdf9c05c300 0x7fdf9c05e7c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fdfb0107b40 tx=0x7fdfac0099e0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:49.570 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.569+0000 7fdfa67fc700 1 -- 172.21.15.58:0/4000073478 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfa004a880 con 0x7fdfb00f7840 2021-11-20T05:37:49.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:49 smithi187 conmon[36358]: cluster 2021-11-20T05:37:48.289855+0000 mgr.smithi058.zagdlk (mgr 2021-11-20T05:37:49.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:49 smithi187 conmon[36358]: .14164) 303 : cluster [DBG] pgmap v277: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:49.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:49 smithi058 conmon[33537]: cluster 2021-11-20T05:37:48.289855+0000 mgr.smithi058.zagdlk (mgr.14164) 303 : cluster [DBG] pgmap v277: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:49.758 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.762+0000 7fdfb6ac2700 1 -- 172.21.15.58:0/4000073478 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfb0045ad0 con 0x7fdfb00f7840 2021-11-20T05:37:49.759 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.763+0000 7fdfa67fc700 1 -- 172.21.15.58:0/4000073478 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2570 (secure 0 0 0) 0x7fdfa004dea0 con 0x7fdfb00f7840 2021-11-20T05:37:49.760 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:49.760 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:45.403981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24423,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24423":{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:49.761 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.764+0000 7fdfb6ac2700 1 -- 172.21.15.58:0/4000073478 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdf9c05c300 msgr2=0x7fdf9c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:49.762 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.764+0000 7fdfb6ac2700 1 --2- 172.21.15.58:0/4000073478 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdf9c05c300 0x7fdf9c05e7c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fdfb0107b40 tx=0x7fdfac0099e0).stop 2021-11-20T05:37:49.762 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.764+0000 7fdfb6ac2700 1 -- 172.21.15.58:0/4000073478 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdfb00f7840 msgr2=0x7fdfb0107270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:49.762 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.764+0000 7fdfb6ac2700 1 --2- 172.21.15.58:0/4000073478 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdfb00f7840 0x7fdfb0107270 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fdfa0009f50 tx=0x7fdfa0009b30).stop 2021-11-20T05:37:49.762 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.764+0000 7fdfb6ac2700 1 -- 172.21.15.58:0/4000073478 shutdown_connections 2021-11-20T05:37:49.763 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.764+0000 7fdfb6ac2700 1 --2- 172.21.15.58:0/4000073478 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fdf9c05c300 0x7fdf9c05e7c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:49.764 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.764+0000 7fdfb6ac2700 1 --2- 172.21.15.58:0/4000073478 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb00f65a0 0x7fdfb0106d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:37:49.764 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.764+0000 7fdfb6ac2700 1 --2- 172.21.15.58:0/4000073478 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fdfb00f7840 0x7fdfb0107270 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:49.765 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.764+0000 7fdfb6ac2700 1 -- 172.21.15.58:0/4000073478 >> 172.21.15.58:0/4000073478 conn(0x7fdfb00f1ce0 msgr2=0x7fdfb00fb290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:49.765 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.765+0000 7fdfb6ac2700 1 -- 172.21.15.58:0/4000073478 shutdown_connections 2021-11-20T05:37:49.765 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:49.765+0000 7fdfb6ac2700 1 -- 172.21.15.58:0/4000073478 wait complete. 2021-11-20T05:37:49.765 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 78 2021-11-20T05:37:49.773 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:45.403981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24423, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24423': {'gid': 24423, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 14, 'addr': '172.21.15.187:6825/2715421480', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 2715421480}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 2715421480}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:49.773 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24423:up:rejoin 2021-11-20T05:37:50.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:50 smithi187 conmon[36358]: audit 2021-11-20T05:37:49.763810+0000 mon.smithi058 (mon.0) 782 2021-11-20T05:37:50.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:50 smithi187 conmon[36358]: : audit [DBG] from='client.? 172.21.15.58:0/4000073478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:50.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:50 smithi058 conmon[33537]: audit 2021-11-20T05:37:49.763810+0000 mon.smithi058 (mon. 2021-11-20T05:37:50.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:50 smithi058 conmon[33537]: 0) 782 : audit [DBG] from='client.? 172.21.15.58:0/4000073478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:50.774 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:50.910 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.911+0000 7fd8020ec700 1 -- 172.21.15.58:0/157374060 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd7fc0f5580 msgr2=0x7fd7fc0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:50.911 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.911+0000 7fd8020ec700 1 --2- 172.21.15.58:0/157374060 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd7fc0f5580 0x7fd7fc0f59a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fd7ec004660 tx=0x7fd7ec009b30).stop 2021-11-20T05:37:50.911 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.911+0000 7fd8020ec700 1 -- 172.21.15.58:0/157374060 shutdown_connections 2021-11-20T05:37:50.912 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.911+0000 7fd8020ec700 1 --2- 172.21.15.58:0/157374060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7fc0f6820 0x7fd7fc0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:50.912 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.911+0000 7fd8020ec700 1 --2- 172.21.15.58:0/157374060 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd7fc0f5580 0x7fd7fc0f59a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:50.912 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.911+0000 7fd8020ec700 1 -- 172.21.15.58:0/157374060 >> 172.21.15.58:0/157374060 conn(0x7fd7fc0f0ce0 msgr2=0x7fd7fc0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:50.912 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.911+0000 7fd8020ec700 1 -- 172.21.15.58:0/157374060 shutdown_connections 2021-11-20T05:37:50.913 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.911+0000 7fd8020ec700 1 -- 172.21.15.58:0/157374060 wait complete. 2021-11-20T05:37:50.913 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.912+0000 7fd8020ec700 1 Processor -- start 2021-11-20T05:37:50.913 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.912+0000 7fd8020ec700 1 -- start start 2021-11-20T05:37:50.913 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.912+0000 7fd8020ec700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7fc0f5580 0x7fd7fc106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:50.914 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.912+0000 7fd8020ec700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd7fc0f6820 0x7fd7fc107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:50.914 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.912+0000 7fd8020ec700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc107940 con 0x7fd7fc0f6820 2021-11-20T05:37:50.914 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.912+0000 7fd8020ec700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc107a80 con 0x7fd7fc0f5580 2021-11-20T05:37:50.914 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.912+0000 7fd7faffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd7fc0f6820 0x7fd7fc107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:50.915 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.912+0000 7fd7faffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd7fc0f6820 0x7fd7fc107390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41872/0 (socket says 172.21.15.58:41872) 2021-11-20T05:37:50.915 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.912+0000 7fd7faffd700 1 -- 172.21.15.58:0/1718480515 learned_addr learned my addr 172.21.15.58:0/1718480515 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:50.915 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.912+0000 7fd7fb7fe700 1 --2- 172.21.15.58:0/1718480515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7fc0f5580 0x7fd7fc106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:50.915 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.912+0000 7fd7faffd700 1 -- 172.21.15.58:0/1718480515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7fc0f5580 msgr2=0x7fd7fc106ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:50.916 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.912+0000 7fd7faffd700 1 --2- 172.21.15.58:0/1718480515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7fc0f5580 0x7fd7fc106ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:50.916 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.912+0000 7fd7faffd700 1 -- 172.21.15.58:0/1718480515 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec005040 con 0x7fd7fc0f6820 2021-11-20T05:37:50.916 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.912+0000 7fd7fb7fe700 1 --2- 172.21.15.58:0/1718480515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7fc0f5580 0x7fd7fc106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:37:50.916 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.913+0000 7fd7faffd700 1 --2- 172.21.15.58:0/1718480515 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd7fc0f6820 0x7fd7fc107390 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fd7f000c530 tx=0x7fd7f0007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:50.916 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.913+0000 7fd7e3fff700 1 -- 172.21.15.58:0/1718480515 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7f000f650 con 0x7fd7fc0f6820 2021-11-20T05:37:50.917 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.913+0000 7fd8020ec700 1 -- 172.21.15.58:0/1718480515 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7fc10c520 con 0x7fd7fc0f6820 2021-11-20T05:37:50.917 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.913+0000 7fd7e3fff700 1 -- 172.21.15.58:0/1718480515 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7f000fc60 con 0x7fd7fc0f6820 2021-11-20T05:37:50.917 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.913+0000 7fd7e3fff700 1 -- 172.21.15.58:0/1718480515 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7f0012d30 con 0x7fd7fc0f6820 2021-11-20T05:37:50.917 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.913+0000 7fd8020ec700 1 -- 172.21.15.58:0/1718480515 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7fc10cae0 con 0x7fd7fc0f6820 2021-11-20T05:37:50.917 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.914+0000 7fd8020ec700 1 -- 172.21.15.58:0/1718480515 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7fc027a00 con 0x7fd7fc0f6820 2021-11-20T05:37:50.918 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.914+0000 7fd7e3fff700 1 -- 172.21.15.58:0/1718480515 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd7f0010070 con 0x7fd7fc0f6820 2021-11-20T05:37:50.918 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.914+0000 7fd7e3fff700 1 --2- 172.21.15.58:0/1718480515 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd7e405c2a0 0x7fd7e405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:50.918 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.915+0000 7fd7e3fff700 1 -- 172.21.15.58:0/1718480515 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fd7f0076050 con 0x7fd7fc0f6820 2021-11-20T05:37:50.918 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.915+0000 7fd7fb7fe700 1 --2- 172.21.15.58:0/1718480515 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd7e405c2a0 0x7fd7e405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:50.919 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.915+0000 7fd7fb7fe700 1 --2- 172.21.15.58:0/1718480515 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd7e405c2a0 0x7fd7e405e760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fd7ec009d60 tx=0x7fd7ec009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:50.919 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:50.919+0000 7fd7e3fff700 1 -- 172.21.15.58:0/1718480515 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd7f00452b0 con 0x7fd7fc0f6820 2021-11-20T05:37:51.111 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:51.115+0000 7fd8020ec700 1 -- 172.21.15.58:0/1718480515 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7fc0fee20 con 0x7fd7fc0f6820 2021-11-20T05:37:51.112 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:51.115+0000 7fd7e3fff700 1 -- 172.21.15.58:0/1718480515 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2570 (secure 0 0 0) 0x7fd7f0048910 con 0x7fd7fc0f6820 2021-11-20T05:37:51.112 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:51.112 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:45.403981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24423,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24423":{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:51.113 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:51.117+0000 7fd8020ec700 1 -- 172.21.15.58:0/1718480515 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd7e405c2a0 msgr2=0x7fd7e405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:51.114 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:51.117+0000 7fd8020ec700 1 --2- 172.21.15.58:0/1718480515 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd7e405c2a0 0x7fd7e405e760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fd7ec009d60 tx=0x7fd7ec009b30).stop 2021-11-20T05:37:51.114 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:51.117+0000 7fd8020ec700 1 -- 172.21.15.58:0/1718480515 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd7fc0f6820 msgr2=0x7fd7fc107390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:51.114 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:51.117+0000 7fd8020ec700 1 --2- 172.21.15.58:0/1718480515 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd7fc0f6820 0x7fd7fc107390 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fd7f000c530 tx=0x7fd7f0007e20).stop 2021-11-20T05:37:51.115 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:51.117+0000 7fd8020ec700 1 -- 172.21.15.58:0/1718480515 shutdown_connections 2021-11-20T05:37:51.115 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:51.117+0000 7fd8020ec700 1 --2- 172.21.15.58:0/1718480515 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd7e405c2a0 0x7fd7e405e760 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:51.115 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:51.117+0000 7fd8020ec700 1 --2- 172.21.15.58:0/1718480515 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd7fc0f5580 0x7fd7fc106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:51.115 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:51.117+0000 7fd8020ec700 1 --2- 172.21.15.58:0/1718480515 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd7fc0f6820 0x7fd7fc107390 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:51.115 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:51.117+0000 7fd8020ec700 1 -- 172.21.15.58:0/1718480515 >> 172.21.15.58:0/1718480515 conn(0x7fd7fc0f0ce0 msgr2=0x7fd7fc0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:51.116 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:51.117+0000 7fd8020ec700 1 -- 172.21.15.58:0/1718480515 shutdown_connections 2021-11-20T05:37:51.116 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:51.117+0000 7fd8020ec700 1 -- 172.21.15.58:0/1718480515 wait complete. 2021-11-20T05:37:51.116 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 78 2021-11-20T05:37:51.125 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:45.403981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24423, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24423': {'gid': 24423, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 14, 'addr': '172.21.15.187:6825/2715421480', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 2715421480}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 2715421480}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:51.126 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24423:up:rejoin 2021-11-20T05:37:51.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:51 smithi187 conmon[36358]: cluster 2021-11-20T05:37:50.290354+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:37:51.770 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:51 smithi187 conmon[36358]: mgr.14164) 304 : cluster [DBG] pgmap v278: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:51.770 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:51 smithi187 conmon[36358]: audit 2021-11-20T05:37:51.116345+0000 mon.smithi058 (mon.0) 783 : 2021-11-20T05:37:51.770 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:51 smithi187 conmon[36358]: audit [DBG] from='client.? 172.21.15.58:0/1718480515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:51.771 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:51 smithi058 conmon[33537]: cluster 2021-11-20T05:37: 2021-11-20T05:37:51.772 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:51 smithi058 conmon[33537]: 50.290354+0000 mgr.smithi058.zagdlk (mgr.14164) 304 : cluster [DBG] pgmap v278: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:51.772 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:51 smithi058 conmon[33537]: audit 2021 2021-11-20T05:37:51.772 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:51 smithi058 conmon[33537]: -11-20T05:37:51.116345+0000 mon.smithi058 (mon.0) 2021-11-20T05:37:51.772 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:51 smithi058 conmon[33537]: 783 : audit [DBG] from='client.? 172.21.15.58:0/1718480515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:52.127 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:52.262 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.263+0000 7f446ad77700 1 -- 172.21.15.58:0/2037380063 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44640f6a70 msgr2=0x7f44640f6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:52.262 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.263+0000 7f446ad77700 1 --2- 172.21.15.58:0/2037380063 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44640f6a70 0x7f44640f6e90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f4460004660 tx=0x7f4460009b30).stop 2021-11-20T05:37:52.262 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.263+0000 7f446ad77700 1 -- 172.21.15.58:0/2037380063 shutdown_connections 2021-11-20T05:37:52.263 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.263+0000 7f446ad77700 1 --2- 172.21.15.58:0/2037380063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44640f7410 0x7f44640f46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:52.263 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.263+0000 7f446ad77700 1 --2- 172.21.15.58:0/2037380063 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44640f6a70 0x7f44640f6e90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:52.263 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.263+0000 7f446ad77700 1 -- 172.21.15.58:0/2037380063 >> 172.21.15.58:0/2037380063 conn(0x7f44640f0360 msgr2=0x7f44640f27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:52.263 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.264+0000 7f446ad77700 1 -- 172.21.15.58:0/2037380063 shutdown_connections 2021-11-20T05:37:52.264 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.264+0000 7f446ad77700 1 -- 172.21.15.58:0/2037380063 wait complete. 2021-11-20T05:37:52.264 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.264+0000 7f446ad77700 1 Processor -- start 2021-11-20T05:37:52.264 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.264+0000 7f446ad77700 1 -- start start 2021-11-20T05:37:52.264 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.264+0000 7f446ad77700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44640f6a70 0x7f4464108f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:52.265 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.264+0000 7f446ad77700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44640f7410 0x7f4464109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:52.265 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.264+0000 7f446ad77700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4464109a00 con 0x7f44640f6a70 2021-11-20T05:37:52.265 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.264+0000 7f446ad77700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4464109b40 con 0x7f44640f7410 2021-11-20T05:37:52.265 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.264+0000 7f4469d75700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44640f6a70 0x7f4464108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:52.265 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.264+0000 7f4469d75700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44640f6a70 0x7f4464108f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41882/0 (socket says 172.21.15.58:41882) 2021-11-20T05:37:52.266 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.264+0000 7f4469d75700 1 -- 172.21.15.58:0/914058067 learned_addr learned my addr 172.21.15.58:0/914058067 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:52.266 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.265+0000 7f4469574700 1 --2- 172.21.15.58:0/914058067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44640f7410 0x7f4464109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:52.266 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.265+0000 7f4469d75700 1 -- 172.21.15.58:0/914058067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44640f7410 msgr2=0x7f4464109450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:52.266 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.265+0000 7f4469d75700 1 --2- 172.21.15.58:0/914058067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44640f7410 0x7f4464109450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:52.267 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.265+0000 7f4469d75700 1 -- 172.21.15.58:0/914058067 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4460005040 con 0x7f44640f6a70 2021-11-20T05:37:52.267 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.265+0000 7f4469d75700 1 --2- 172.21.15.58:0/914058067 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44640f6a70 0x7f4464108f60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f4460009830 tx=0x7f4460009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:52.267 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.265+0000 7f445a7fc700 1 -- 172.21.15.58:0/914058067 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4460018070 con 0x7f44640f6a70 2021-11-20T05:37:52.267 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.265+0000 7f446ad77700 1 -- 172.21.15.58:0/914058067 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446410a480 con 0x7f44640f6a70 2021-11-20T05:37:52.268 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.265+0000 7f445a7fc700 1 -- 172.21.15.58:0/914058067 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4460013410 con 0x7f44640f6a70 2021-11-20T05:37:52.268 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.265+0000 7f445a7fc700 1 -- 172.21.15.58:0/914058067 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f446001c620 con 0x7f44640f6a70 2021-11-20T05:37:52.268 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.265+0000 7f446ad77700 1 -- 172.21.15.58:0/914058067 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f446410a9e0 con 0x7f44640f6a70 2021-11-20T05:37:52.268 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.266+0000 7f446ad77700 1 -- 172.21.15.58:0/914058067 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4464045ad0 con 0x7f44640f6a70 2021-11-20T05:37:52.269 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.266+0000 7f445a7fc700 1 -- 172.21.15.58:0/914058067 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f446001c8e0 con 0x7f44640f6a70 2021-11-20T05:37:52.270 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.267+0000 7f445a7fc700 1 --2- 172.21.15.58:0/914058067 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f445005c300 0x7f445005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:52.270 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.267+0000 7f445a7fc700 1 -- 172.21.15.58:0/914058067 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f44600781c0 con 0x7f44640f6a70 2021-11-20T05:37:52.270 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.267+0000 7f4469574700 1 --2- 172.21.15.58:0/914058067 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f445005c300 0x7f445005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:52.270 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.267+0000 7f4469574700 1 --2- 172.21.15.58:0/914058067 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f445005c300 0x7f445005e7c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f4454007fd0 tx=0x7f4454009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:52.271 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.271+0000 7f445a7fc700 1 -- 172.21.15.58:0/914058067 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f446004aa20 con 0x7f44640f6a70 2021-11-20T05:37:52.468 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.472+0000 7f446ad77700 1 -- 172.21.15.58:0/914058067 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4464101060 con 0x7f44640f6a70 2021-11-20T05:37:52.468 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.472+0000 7f445a7fc700 1 -- 172.21.15.58:0/914058067 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2570 (secure 0 0 0) 0x7f446004e040 con 0x7f44640f6a70 2021-11-20T05:37:52.469 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:52.469 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:45.403981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24423,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24423":{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:52.471 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.474+0000 7f446ad77700 1 -- 172.21.15.58:0/914058067 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f445005c300 msgr2=0x7f445005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:52.471 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.474+0000 7f446ad77700 1 --2- 172.21.15.58:0/914058067 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f445005c300 0x7f445005e7c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f4454007fd0 tx=0x7f4454009040).stop 2021-11-20T05:37:52.472 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.474+0000 7f446ad77700 1 -- 172.21.15.58:0/914058067 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44640f6a70 msgr2=0x7f4464108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:52.472 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.474+0000 7f446ad77700 1 --2- 172.21.15.58:0/914058067 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44640f6a70 0x7f4464108f60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f4460009830 tx=0x7f4460009b30).stop 2021-11-20T05:37:52.472 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.474+0000 7f446ad77700 1 -- 172.21.15.58:0/914058067 shutdown_connections 2021-11-20T05:37:52.472 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.474+0000 7f446ad77700 1 --2- 172.21.15.58:0/914058067 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f445005c300 0x7f445005e7c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:52.473 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.474+0000 7f446ad77700 1 --2- 172.21.15.58:0/914058067 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44640f6a70 0x7f4464108f60 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:52.473 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.474+0000 7f446ad77700 1 --2- 172.21.15.58:0/914058067 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44640f7410 0x7f4464109450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:52.473 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.474+0000 7f446ad77700 1 -- 172.21.15.58:0/914058067 >> 172.21.15.58:0/914058067 conn(0x7f44640f0360 msgr2=0x7f44640fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:52.473 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.474+0000 7f446ad77700 1 -- 172.21.15.58:0/914058067 shutdown_connections 2021-11-20T05:37:52.473 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:52.474+0000 7f446ad77700 1 -- 172.21.15.58:0/914058067 wait complete. 2021-11-20T05:37:52.474 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 78 2021-11-20T05:37:52.482 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:45.403981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24423, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24423': {'gid': 24423, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 14, 'addr': '172.21.15.187:6825/2715421480', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 2715421480}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 2715421480}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:52.483 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24423:up:rejoin 2021-11-20T05:37:53.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:53 smithi058 conmon[33537]: cluster 2021-11-20T05:37:52.290816+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:37:53.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:53 smithi058 conmon[33537]: ) 305 : cluster [DBG] pgmap v279: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:53.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:53 smithi058 conmon[33537]: audit 2021-11-20T05:37:52.473111+0000 mon.smithi058 (mon.0) 784 : audit [DBG] from='client.? 172.21.15.58:0/914058067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:53.485 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:53.619 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.621+0000 7f1f99268700 1 -- 172.21.15.58:0/4280755921 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1f940f6820 msgr2=0x7f1f940f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:53.620 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.621+0000 7f1f99268700 1 --2- 172.21.15.58:0/4280755921 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1f940f6820 0x7f1f940f6c80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f1f88004660 tx=0x7f1f88009b30).stop 2021-11-20T05:37:53.620 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.621+0000 7f1f99268700 1 -- 172.21.15.58:0/4280755921 shutdown_connections 2021-11-20T05:37:53.620 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.621+0000 7f1f99268700 1 --2- 172.21.15.58:0/4280755921 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1f940f6820 0x7f1f940f6c80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:53.620 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.621+0000 7f1f99268700 1 --2- 172.21.15.58:0/4280755921 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f940f5580 0x7f1f940f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:53.621 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.621+0000 7f1f99268700 1 -- 172.21.15.58:0/4280755921 >> 172.21.15.58:0/4280755921 conn(0x7f1f940f0ce0 msgr2=0x7f1f940f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:53.621 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.621+0000 7f1f99268700 1 -- 172.21.15.58:0/4280755921 shutdown_connections 2021-11-20T05:37:53.621 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.621+0000 7f1f99268700 1 -- 172.21.15.58:0/4280755921 wait complete. 2021-11-20T05:37:53.621 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.622+0000 7f1f99268700 1 Processor -- start 2021-11-20T05:37:53.622 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.622+0000 7f1f99268700 1 -- start start 2021-11-20T05:37:53.622 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.622+0000 7f1f99268700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1f940f5580 0x7f1f94106e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:53.622 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.622+0000 7f1f99268700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f940f6820 0x7f1f94107360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:53.622 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.622+0000 7f1f99268700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f94107910 con 0x7f1f940f5580 2021-11-20T05:37:53.623 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.622+0000 7f1f99268700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f94107a50 con 0x7f1f940f6820 2021-11-20T05:37:53.623 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.622+0000 7f1f92ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1f940f5580 0x7f1f94106e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:53.624 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.622+0000 7f1f92ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1f940f5580 0x7f1f94106e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41894/0 (socket says 172.21.15.58:41894) 2021-11-20T05:37:53.624 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.622+0000 7f1f92ffd700 1 -- 172.21.15.58:0/1318217083 learned_addr learned my addr 172.21.15.58:0/1318217083 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:53.624 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.623+0000 7f1f927fc700 1 --2- 172.21.15.58:0/1318217083 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f940f6820 0x7f1f94107360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:53.625 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.623+0000 7f1f92ffd700 1 -- 172.21.15.58:0/1318217083 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f940f6820 msgr2=0x7f1f94107360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:53.625 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.623+0000 7f1f92ffd700 1 --2- 172.21.15.58:0/1318217083 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f940f6820 0x7f1f94107360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:53.625 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.623+0000 7f1f92ffd700 1 -- 172.21.15.58:0/1318217083 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f88005040 con 0x7f1f940f5580 2021-11-20T05:37:53.625 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.623+0000 7f1f927fc700 1 --2- 172.21.15.58:0/1318217083 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f940f6820 0x7f1f94107360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:37:53.626 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.623+0000 7f1f92ffd700 1 --2- 172.21.15.58:0/1318217083 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1f940f5580 0x7f1f94106e70 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f1f8400c790 tx=0x7f1f84007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:53.626 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.623+0000 7f1f7b7fe700 1 -- 172.21.15.58:0/1318217083 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f8400e680 con 0x7f1f940f5580 2021-11-20T05:37:53.626 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.623+0000 7f1f99268700 1 -- 172.21.15.58:0/1318217083 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f9410c4f0 con 0x7f1f940f5580 2021-11-20T05:37:53.626 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.623+0000 7f1f7b7fe700 1 -- 172.21.15.58:0/1318217083 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f8400ec90 con 0x7f1f940f5580 2021-11-20T05:37:53.626 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.623+0000 7f1f7b7fe700 1 -- 172.21.15.58:0/1318217083 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f84012cb0 con 0x7f1f940f5580 2021-11-20T05:37:53.627 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.623+0000 7f1f99268700 1 -- 172.21.15.58:0/1318217083 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9410cab0 con 0x7f1f940f5580 2021-11-20T05:37:53.628 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.624+0000 7f1f99268700 1 -- 172.21.15.58:0/1318217083 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f940fdd10 con 0x7f1f940f5580 2021-11-20T05:37:53.628 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.625+0000 7f1f7b7fe700 1 -- 172.21.15.58:0/1318217083 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1f84010070 con 0x7f1f940f5580 2021-11-20T05:37:53.628 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.625+0000 7f1f7b7fe700 1 --2- 172.21.15.58:0/1318217083 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1f7c05c2a0 0x7f1f7c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:53.629 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.625+0000 7f1f7b7fe700 1 -- 172.21.15.58:0/1318217083 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f1f84076430 con 0x7f1f940f5580 2021-11-20T05:37:53.629 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.625+0000 7f1f927fc700 1 --2- 172.21.15.58:0/1318217083 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1f7c05c2a0 0x7f1f7c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:53.629 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.626+0000 7f1f927fc700 1 --2- 172.21.15.58:0/1318217083 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1f7c05c2a0 0x7f1f7c05e760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f1f88013580 tx=0x7f1f88009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:53.629 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.629+0000 7f1f7b7fe700 1 -- 172.21.15.58:0/1318217083 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f84048d20 con 0x7f1f940f5580 2021-11-20T05:37:53.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:53 smithi187 conmon[36358]: cluster 2021-11-20T05:37:52.290816+0000 mgr.smithi058.zagdlk (mgr.14164) 305 : cluster [DBG] pgmap v279: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:37:53.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:53 smithi187 conmon[36358]: audit 2021-11-20T05:37:52.473111+0000 mon.smithi058 (mon.0) 784 : audit [DBG] from='client.? 172.21.15.58:0/914058067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:53.821 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.825+0000 7f1f99268700 1 -- 172.21.15.58:0/1318217083 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f9410d150 con 0x7f1f940f5580 2021-11-20T05:37:53.822 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.825+0000 7f1f7b7fe700 1 -- 172.21.15.58:0/1318217083 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2570 (secure 0 0 0) 0x7f1f8404c340 con 0x7f1f940f5580 2021-11-20T05:37:53.822 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:53.823 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:45.403981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24423,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24423":{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:53.824 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.827+0000 7f1f99268700 1 -- 172.21.15.58:0/1318217083 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1f7c05c2a0 msgr2=0x7f1f7c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:53.824 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.827+0000 7f1f99268700 1 --2- 172.21.15.58:0/1318217083 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1f7c05c2a0 0x7f1f7c05e760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f1f88013580 tx=0x7f1f88009b30).stop 2021-11-20T05:37:53.824 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.827+0000 7f1f99268700 1 -- 172.21.15.58:0/1318217083 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1f940f5580 msgr2=0x7f1f94106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:53.825 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.827+0000 7f1f99268700 1 --2- 172.21.15.58:0/1318217083 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1f940f5580 0x7f1f94106e70 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f1f8400c790 tx=0x7f1f84007d90).stop 2021-11-20T05:37:53.825 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.827+0000 7f1f99268700 1 -- 172.21.15.58:0/1318217083 shutdown_connections 2021-11-20T05:37:53.825 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.827+0000 7f1f99268700 1 --2- 172.21.15.58:0/1318217083 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1f7c05c2a0 0x7f1f7c05e760 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:53.825 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.827+0000 7f1f99268700 1 --2- 172.21.15.58:0/1318217083 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1f940f5580 0x7f1f94106e70 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:53.826 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.827+0000 7f1f99268700 1 --2- 172.21.15.58:0/1318217083 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f940f6820 0x7f1f94107360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:53.826 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.827+0000 7f1f99268700 1 -- 172.21.15.58:0/1318217083 >> 172.21.15.58:0/1318217083 conn(0x7f1f940f0ce0 msgr2=0x7f1f94060240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:53.826 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.827+0000 7f1f99268700 1 -- 172.21.15.58:0/1318217083 shutdown_connections 2021-11-20T05:37:53.826 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:53.827+0000 7f1f99268700 1 -- 172.21.15.58:0/1318217083 wait complete. 2021-11-20T05:37:53.827 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 78 2021-11-20T05:37:53.835 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:45.403981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24423, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24423': {'gid': 24423, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 14, 'addr': '172.21.15.187:6825/2715421480', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 2715421480}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 2715421480}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:53.836 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24423:up:rejoin 2021-11-20T05:37:54.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:54 smithi187 conmon[36358]: audit 2021-11-20T05:37:53.826510+0000 mon.smithi058 (mon.0) 785 : 2021-11-20T05:37:54.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:54 smithi187 conmon[36358]: audit [DBG] from='client.? 172.21.15.58:0/1318217083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:54.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:54 smithi058 conmon[33537]: audit 2021-11-20T05:37:53.826510+0000 mon.smithi058 (mon.0 2021-11-20T05:37:54.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:54 smithi058 conmon[33537]: ) 785 : audit [DBG] from='client.? 172.21.15.58:0/1318217083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:54.837 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:54.971 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.973+0000 7f7db1071700 1 -- 172.21.15.58:0/4023645514 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7dac0f7960 msgr2=0x7f7dac0f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:54.972 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.973+0000 7f7db1071700 1 --2- 172.21.15.58:0/4023645514 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7dac0f7960 0x7f7dac0f7dc0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f7da0004660 tx=0x7f7da0009b30).stop 2021-11-20T05:37:54.972 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.973+0000 7f7db1071700 1 -- 172.21.15.58:0/4023645514 shutdown_connections 2021-11-20T05:37:54.972 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.973+0000 7f7db1071700 1 --2- 172.21.15.58:0/4023645514 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7dac0f7960 0x7f7dac0f7dc0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:54.973 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.973+0000 7f7db1071700 1 --2- 172.21.15.58:0/4023645514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dac0f66c0 0x7f7dac0f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:54.973 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.973+0000 7f7db1071700 1 -- 172.21.15.58:0/4023645514 >> 172.21.15.58:0/4023645514 conn(0x7f7dac0f1e40 msgr2=0x7f7dac0f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:54.973 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.973+0000 7f7db1071700 1 -- 172.21.15.58:0/4023645514 shutdown_connections 2021-11-20T05:37:54.973 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.974+0000 7f7db1071700 1 -- 172.21.15.58:0/4023645514 wait complete. 2021-11-20T05:37:54.973 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.974+0000 7f7db1071700 1 Processor -- start 2021-11-20T05:37:54.974 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.974+0000 7f7db1071700 1 -- start start 2021-11-20T05:37:54.974 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.974+0000 7f7db1071700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7dac0f66c0 0x7f7dac106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:54.974 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.974+0000 7f7db1071700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dac0f7960 0x7f7dac1073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:54.974 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.974+0000 7f7db1071700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac107950 con 0x7f7dac0f66c0 2021-11-20T05:37:54.975 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.974+0000 7f7db1071700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac107a90 con 0x7f7dac0f7960 2021-11-20T05:37:54.975 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.974+0000 7f7daad9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7dac0f66c0 0x7f7dac106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:54.975 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.975+0000 7f7daad9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7dac0f66c0 0x7f7dac106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41904/0 (socket says 172.21.15.58:41904) 2021-11-20T05:37:54.975 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.975+0000 7f7daad9d700 1 -- 172.21.15.58:0/2059883598 learned_addr learned my addr 172.21.15.58:0/2059883598 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:54.976 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.975+0000 7f7daa59c700 1 --2- 172.21.15.58:0/2059883598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dac0f7960 0x7f7dac1073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:54.976 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.975+0000 7f7daad9d700 1 -- 172.21.15.58:0/2059883598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dac0f7960 msgr2=0x7f7dac1073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:54.976 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.975+0000 7f7daad9d700 1 --2- 172.21.15.58:0/2059883598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dac0f7960 0x7f7dac1073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:54.976 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.975+0000 7f7daad9d700 1 -- 172.21.15.58:0/2059883598 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da0005040 con 0x7f7dac0f66c0 2021-11-20T05:37:54.976 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.975+0000 7f7daa59c700 1 --2- 172.21.15.58:0/2059883598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dac0f7960 0x7f7dac1073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:37:54.977 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.975+0000 7f7daad9d700 1 --2- 172.21.15.58:0/2059883598 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7dac0f66c0 0x7f7dac106eb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f7d9c002700 tx=0x7f7d9c009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:54.977 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.975+0000 7f7d937fe700 1 -- 172.21.15.58:0/2059883598 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d9c009320 con 0x7f7dac0f66c0 2021-11-20T05:37:54.977 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.975+0000 7f7d937fe700 1 -- 172.21.15.58:0/2059883598 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d9c00f040 con 0x7f7dac0f66c0 2021-11-20T05:37:54.977 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.975+0000 7f7db1071700 1 -- 172.21.15.58:0/2059883598 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dac10c530 con 0x7f7dac0f66c0 2021-11-20T05:37:54.978 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.975+0000 7f7d937fe700 1 -- 172.21.15.58:0/2059883598 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d9c014900 con 0x7f7dac0f66c0 2021-11-20T05:37:54.978 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.975+0000 7f7db1071700 1 -- 172.21.15.58:0/2059883598 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dac10caf0 con 0x7f7dac0f66c0 2021-11-20T05:37:54.979 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.976+0000 7f7db1071700 1 -- 172.21.15.58:0/2059883598 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dac027a00 con 0x7f7dac0f66c0 2021-11-20T05:37:54.979 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.977+0000 7f7d937fe700 1 -- 172.21.15.58:0/2059883598 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7d9c009480 con 0x7f7dac0f66c0 2021-11-20T05:37:54.980 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.977+0000 7f7d937fe700 1 --2- 172.21.15.58:0/2059883598 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7d9405c2a0 0x7f7d9405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:54.980 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.977+0000 7f7d937fe700 1 -- 172.21.15.58:0/2059883598 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f7d9c076680 con 0x7f7dac0f66c0 2021-11-20T05:37:54.980 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.977+0000 7f7daa59c700 1 --2- 172.21.15.58:0/2059883598 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7d9405c2a0 0x7f7d9405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:54.980 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.977+0000 7f7daa59c700 1 --2- 172.21.15.58:0/2059883598 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7d9405c2a0 0x7f7d9405e760 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f7da0005010 tx=0x7f7da0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:54.980 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:54.981+0000 7f7d937fe700 1 -- 172.21.15.58:0/2059883598 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d9c048f40 con 0x7f7dac0f66c0 2021-11-20T05:37:55.174 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:55.178+0000 7f7db1071700 1 -- 172.21.15.58:0/2059883598 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7dac0fee50 con 0x7f7dac0f66c0 2021-11-20T05:37:55.174 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:55.178+0000 7f7d937fe700 1 -- 172.21.15.58:0/2059883598 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2570 (secure 0 0 0) 0x7f7d9c04c560 con 0x7f7dac0f66c0 2021-11-20T05:37:55.175 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:55.175 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:45.403981+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24423,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24423":{"gid":24423,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.187:6825/2715421480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2715421480},{"type":"v1","addr":"172.21.15.187:6825","nonce":2715421480}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:55.176 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:55.180+0000 7f7db1071700 1 -- 172.21.15.58:0/2059883598 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7d9405c2a0 msgr2=0x7f7d9405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:55.177 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:55.180+0000 7f7db1071700 1 --2- 172.21.15.58:0/2059883598 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7d9405c2a0 0x7f7d9405e760 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f7da0005010 tx=0x7f7da0009b30).stop 2021-11-20T05:37:55.177 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:55.180+0000 7f7db1071700 1 -- 172.21.15.58:0/2059883598 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7dac0f66c0 msgr2=0x7f7dac106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:55.177 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:55.180+0000 7f7db1071700 1 --2- 172.21.15.58:0/2059883598 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7dac0f66c0 0x7f7dac106eb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f7d9c002700 tx=0x7f7d9c009d80).stop 2021-11-20T05:37:55.177 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:55.180+0000 7f7db1071700 1 -- 172.21.15.58:0/2059883598 shutdown_connections 2021-11-20T05:37:55.178 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:55.180+0000 7f7db1071700 1 --2- 172.21.15.58:0/2059883598 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7d9405c2a0 0x7f7d9405e760 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:55.178 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:55.180+0000 7f7db1071700 1 --2- 172.21.15.58:0/2059883598 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7dac0f66c0 0x7f7dac106eb0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:55.178 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:55.180+0000 7f7db1071700 1 --2- 172.21.15.58:0/2059883598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7dac0f7960 0x7f7dac1073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:55.178 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:55.180+0000 7f7db1071700 1 -- 172.21.15.58:0/2059883598 >> 172.21.15.58:0/2059883598 conn(0x7f7dac0f1e40 msgr2=0x7f7dac0fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:55.179 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:55.180+0000 7f7db1071700 1 -- 172.21.15.58:0/2059883598 shutdown_connections 2021-11-20T05:37:55.179 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:55.180+0000 7f7db1071700 1 -- 172.21.15.58:0/2059883598 wait complete. 2021-11-20T05:37:55.180 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 78 2021-11-20T05:37:55.188 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:45.403981+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24423, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24423': {'gid': 24423, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 14, 'addr': '172.21.15.187:6825/2715421480', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 2715421480}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 2715421480}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:55.189 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24423:up:rejoin 2021-11-20T05:37:55.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:55 smithi187 conmon[36358]: cluster 2021-11-20T05:37:54.291309+0000 mgr.smithi058.zagdlk (mgr.14164) 306 : cluster [DBG] pgmap v280: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:37:55.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:55 smithi187 conmon[36358]: audit 2021-11-20T05:37:55.179228+0000 mon.smithi058 (mon.0) 786 : audit [DBG] from='client.? 172.21.15.58:0/2059883598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:55.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:55 smithi058 conmon[33537]: cluster 2021-11-20T05:37:54.291309+0000 mgr.smithi058.zagdlk (mgr.14164) 306 : cluster [DBG] pgmap v280: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:37:55.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:55 smithi058 conmon[33537]: audit 2021-11-20T05:37:55.179228+0000 mon.smithi058 (mon.0) 786 : audit [DBG] from='client.? 172.21.15.58:0/2059883598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:56.191 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:56.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.323+0000 7f1bbea1e700 1 -- 172.21.15.58:0/1035671938 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1bb80f6820 msgr2=0x7f1bb80f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:56.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.323+0000 7f1bbea1e700 1 --2- 172.21.15.58:0/1035671938 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1bb80f6820 0x7f1bb80f6c80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f1bac004660 tx=0x7f1bac009b30).stop 2021-11-20T05:37:56.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.323+0000 7f1bbea1e700 1 -- 172.21.15.58:0/1035671938 shutdown_connections 2021-11-20T05:37:56.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.323+0000 7f1bbea1e700 1 --2- 172.21.15.58:0/1035671938 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1bb80f6820 0x7f1bb80f6c80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:56.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.324+0000 7f1bbea1e700 1 --2- 172.21.15.58:0/1035671938 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bb80f5580 0x7f1bb80f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:56.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.324+0000 7f1bbea1e700 1 -- 172.21.15.58:0/1035671938 >> 172.21.15.58:0/1035671938 conn(0x7f1bb80f0ce0 msgr2=0x7f1bb80f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:56.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.324+0000 7f1bbea1e700 1 -- 172.21.15.58:0/1035671938 shutdown_connections 2021-11-20T05:37:56.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.324+0000 7f1bbea1e700 1 -- 172.21.15.58:0/1035671938 wait complete. 2021-11-20T05:37:56.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.324+0000 7f1bbea1e700 1 Processor -- start 2021-11-20T05:37:56.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.324+0000 7f1bbea1e700 1 -- start start 2021-11-20T05:37:56.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.324+0000 7f1bbea1e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bb80f5580 0x7f1bb8060510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:56.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.324+0000 7f1bbea1e700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1bb80f6820 0x7f1bb810bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:56.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.324+0000 7f1bbea1e700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb810c100 con 0x7f1bb80f6820 2021-11-20T05:37:56.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.324+0000 7f1bbea1e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb810c240 con 0x7f1bb80f5580 2021-11-20T05:37:56.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.325+0000 7f1bb77fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1bb80f6820 0x7f1bb810bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:56.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.325+0000 7f1bb77fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1bb80f6820 0x7f1bb810bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41916/0 (socket says 172.21.15.58:41916) 2021-11-20T05:37:56.326 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.325+0000 7f1bb77fe700 1 -- 172.21.15.58:0/3316137248 learned_addr learned my addr 172.21.15.58:0/3316137248 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:56.326 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.325+0000 7f1bb7fff700 1 --2- 172.21.15.58:0/3316137248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bb80f5580 0x7f1bb8060510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:56.326 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.325+0000 7f1bb77fe700 1 -- 172.21.15.58:0/3316137248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bb80f5580 msgr2=0x7f1bb8060510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:56.326 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.325+0000 7f1bb77fe700 1 --2- 172.21.15.58:0/3316137248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bb80f5580 0x7f1bb8060510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:56.327 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.325+0000 7f1bb77fe700 1 -- 172.21.15.58:0/3316137248 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bac005040 con 0x7f1bb80f6820 2021-11-20T05:37:56.327 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.325+0000 7f1bb77fe700 1 --2- 172.21.15.58:0/3316137248 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1bb80f6820 0x7f1bb810bc10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f1bac009870 tx=0x7f1bac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:56.327 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.325+0000 7f1bb4ff9700 1 -- 172.21.15.58:0/3316137248 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bac019070 con 0x7f1bb80f6820 2021-11-20T05:37:56.327 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.325+0000 7f1bb4ff9700 1 -- 172.21.15.58:0/3316137248 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1bac007e70 con 0x7f1bb80f6820 2021-11-20T05:37:56.327 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.325+0000 7f1bbea1e700 1 -- 172.21.15.58:0/3316137248 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb810c440 con 0x7f1bb80f6820 2021-11-20T05:37:56.328 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.325+0000 7f1bb4ff9700 1 -- 172.21.15.58:0/3316137248 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bac014c90 con 0x7f1bb80f6820 2021-11-20T05:37:56.328 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.325+0000 7f1bbea1e700 1 -- 172.21.15.58:0/3316137248 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb810c9a0 con 0x7f1bb80f6820 2021-11-20T05:37:56.329 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.326+0000 7f1bbea1e700 1 -- 172.21.15.58:0/3316137248 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb8027a00 con 0x7f1bb80f6820 2021-11-20T05:37:56.329 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.327+0000 7f1bb4ff9700 1 -- 172.21.15.58:0/3316137248 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1bac025070 con 0x7f1bb80f6820 2021-11-20T05:37:56.330 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.327+0000 7f1bb4ff9700 1 --2- 172.21.15.58:0/3316137248 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1ba005c2a0 0x7f1ba005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:56.330 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.327+0000 7f1bb4ff9700 1 -- 172.21.15.58:0/3316137248 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1bac0773b0 con 0x7f1bb80f6820 2021-11-20T05:37:56.330 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.327+0000 7f1bb7fff700 1 --2- 172.21.15.58:0/3316137248 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1ba005c2a0 0x7f1ba005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:56.330 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.328+0000 7f1bb7fff700 1 --2- 172.21.15.58:0/3316137248 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1ba005c2a0 0x7f1ba005e760 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f1ba8009fd0 tx=0x7f1ba800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:56.331 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.331+0000 7f1bb4ff9700 1 -- 172.21.15.58:0/3316137248 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1bac049c10 con 0x7f1bb80f6820 2021-11-20T05:37:56.527 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.531+0000 7f1bbea1e700 1 -- 172.21.15.58:0/3316137248 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bb80fdd10 con 0x7f1bb80f6820 2021-11-20T05:37:56.527 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.531+0000 7f1bb4ff9700 1 -- 172.21.15.58:0/3316137248 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2570 (secure 0 0 0) 0x7f1bac04d230 con 0x7f1bb80f6820 2021-11-20T05:37:56.528 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:56.529 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":79}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:56.448461+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24505,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24505":{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":80,"state":"up:replay","state_seq":3,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:56.531 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.533+0000 7f1bbea1e700 1 -- 172.21.15.58:0/3316137248 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1ba005c2a0 msgr2=0x7f1ba005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:56.531 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.533+0000 7f1bbea1e700 1 --2- 172.21.15.58:0/3316137248 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1ba005c2a0 0x7f1ba005e760 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f1ba8009fd0 tx=0x7f1ba800b040).stop 2021-11-20T05:37:56.531 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.533+0000 7f1bbea1e700 1 -- 172.21.15.58:0/3316137248 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1bb80f6820 msgr2=0x7f1bb810bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:56.532 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.533+0000 7f1bbea1e700 1 --2- 172.21.15.58:0/3316137248 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1bb80f6820 0x7f1bb810bc10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f1bac009870 tx=0x7f1bac00b040).stop 2021-11-20T05:37:56.533 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.533+0000 7f1bbea1e700 1 -- 172.21.15.58:0/3316137248 shutdown_connections 2021-11-20T05:37:56.533 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.533+0000 7f1bbea1e700 1 --2- 172.21.15.58:0/3316137248 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1ba005c2a0 0x7f1ba005e760 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:56.533 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.533+0000 7f1bbea1e700 1 --2- 172.21.15.58:0/3316137248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bb80f5580 0x7f1bb8060510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:56.534 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.533+0000 7f1bbea1e700 1 --2- 172.21.15.58:0/3316137248 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1bb80f6820 0x7f1bb810bc10 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:56.535 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.533+0000 7f1bbea1e700 1 -- 172.21.15.58:0/3316137248 >> 172.21.15.58:0/3316137248 conn(0x7f1bb80f0ce0 msgr2=0x7f1bb80fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:56.535 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.533+0000 7f1bbea1e700 1 -- 172.21.15.58:0/3316137248 shutdown_connections 2021-11-20T05:37:56.535 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:56.533+0000 7f1bbea1e700 1 -- 172.21.15.58:0/3316137248 wait complete. 2021-11-20T05:37:56.536 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 80 2021-11-20T05:37:56.540 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:56 smithi058 conmon[33537]: cluster 2021-11-20T05:37:56.169490+0000 mon.smithi058 (mon 2021-11-20T05:37:56.540 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:56 smithi058 conmon[33537]: .0) 787 : cluster [INF] daemon mds.cephfs.smithi187.wxduhu restarted 2021-11-20T05:37:56.540 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:56 smithi058 conmon[33537]: cluster 2021-11-20T05:37:56.173347+0000 mon.smithi058 (mon.0) 788 : cluster [DBG] osdmap e55: 6 total, 6 up, 6 in 2021-11-20T05:37:56.541 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:56.448461+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24505, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24505': {'gid': 24505, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 80, 'state': 'up:replay', 'state_seq': 3, 'addr': '172.21.15.187:6827/96302868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 96302868}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 96302868}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:56.542 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24505:up:replay 2021-11-20T05:37:56.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:56 smithi187 conmon[36358]: cluster 2021-11-20T05:37:56.169490+0000 mon.smithi058 (mon 2021-11-20T05:37:56.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:56 smithi187 conmon[36358]: .0) 787 : cluster [INF] daemon mds.cephfs.smithi187.wxduhu restarted 2021-11-20T05:37:56.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:56 smithi187 conmon[36358]: cluster 2021-11-20T05:37:56.173347+0000 mon.smithi058 (mon.0 2021-11-20T05:37:56.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:56 smithi187 conmon[36358]: ) 788 : cluster [DBG] osdmap e55: 6 total, 6 up, 6 in 2021-11-20T05:37:57.543 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:57.678 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.678+0000 7f683b59e700 1 -- 172.21.15.58:0/2198135200 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f683c0f54d0 msgr2=0x7f683c0f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:57.679 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.678+0000 7f683b59e700 1 --2- 172.21.15.58:0/2198135200 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f683c0f54d0 0x7f683c0f58f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f682c004660 tx=0x7f682c009b30).stop 2021-11-20T05:37:57.679 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.679+0000 7f683b59e700 1 -- 172.21.15.58:0/2198135200 shutdown_connections 2021-11-20T05:37:57.679 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.679+0000 7f683b59e700 1 --2- 172.21.15.58:0/2198135200 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f683c0f6770 0x7f683c0f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:57.679 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.679+0000 7f683b59e700 1 --2- 172.21.15.58:0/2198135200 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f683c0f54d0 0x7f683c0f58f0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:57.680 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.679+0000 7f683b59e700 1 -- 172.21.15.58:0/2198135200 >> 172.21.15.58:0/2198135200 conn(0x7f683c0f0c30 msgr2=0x7f683c0f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:57.680 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.679+0000 7f683b59e700 1 -- 172.21.15.58:0/2198135200 shutdown_connections 2021-11-20T05:37:57.681 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.679+0000 7f683b59e700 1 -- 172.21.15.58:0/2198135200 wait complete. 2021-11-20T05:37:57.681 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.679+0000 7f683b59e700 1 Processor -- start 2021-11-20T05:37:57.681 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.679+0000 7f683b59e700 1 -- start start 2021-11-20T05:37:57.682 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.679+0000 7f683b59e700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f683c0f54d0 0x7f683c109aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:57.682 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.679+0000 7f683b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f683c0f6770 0x7f683c10bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:57.682 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.679+0000 7f683b59e700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c060520 con 0x7f683c0f54d0 2021-11-20T05:37:57.682 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.679+0000 7f683b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c060660 con 0x7f683c0f6770 2021-11-20T05:37:57.683 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.679+0000 7f683a59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f683c0f54d0 0x7f683c109aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:57.683 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.680+0000 7f683a59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f683c0f54d0 0x7f683c109aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41924/0 (socket says 172.21.15.58:41924) 2021-11-20T05:37:57.683 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.680+0000 7f683a59c700 1 -- 172.21.15.58:0/1530265201 learned_addr learned my addr 172.21.15.58:0/1530265201 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:57.683 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.680+0000 7f6839d9b700 1 --2- 172.21.15.58:0/1530265201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f683c0f6770 0x7f683c10bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:57.684 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.680+0000 7f683a59c700 1 -- 172.21.15.58:0/1530265201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f683c0f6770 msgr2=0x7f683c10bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:57.684 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.680+0000 7f683a59c700 1 --2- 172.21.15.58:0/1530265201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f683c0f6770 0x7f683c10bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:57.684 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.680+0000 7f683a59c700 1 -- 172.21.15.58:0/1530265201 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682c005040 con 0x7f683c0f54d0 2021-11-20T05:37:57.684 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.680+0000 7f6839d9b700 1 --2- 172.21.15.58:0/1530265201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f683c0f6770 0x7f683c10bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:37:57.685 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.680+0000 7f683a59c700 1 --2- 172.21.15.58:0/1530265201 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f683c0f54d0 0x7f683c109aa0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f682c004e30 tx=0x7f682c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:57.686 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.680+0000 7f682affd700 1 -- 172.21.15.58:0/1530265201 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f682c018070 con 0x7f683c0f54d0 2021-11-20T05:37:57.686 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.680+0000 7f682affd700 1 -- 172.21.15.58:0/1530265201 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f682c013410 con 0x7f683c0f54d0 2021-11-20T05:37:57.686 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.680+0000 7f683b59e700 1 -- 172.21.15.58:0/1530265201 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f683c10c4f0 con 0x7f683c0f54d0 2021-11-20T05:37:57.686 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.680+0000 7f682affd700 1 -- 172.21.15.58:0/1530265201 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f682c01c3c0 con 0x7f683c0f54d0 2021-11-20T05:37:57.687 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.680+0000 7f683b59e700 1 -- 172.21.15.58:0/1530265201 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f683c10ca20 con 0x7f683c0f54d0 2021-11-20T05:37:57.687 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.681+0000 7f683b59e700 1 -- 172.21.15.58:0/1530265201 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f683c0f48c0 con 0x7f683c0f54d0 2021-11-20T05:37:57.687 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.681+0000 7f682affd700 1 -- 172.21.15.58:0/1530265201 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f682c01d420 con 0x7f683c0f54d0 2021-11-20T05:37:57.689 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.682+0000 7f682affd700 1 --2- 172.21.15.58:0/1530265201 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f682405c300 0x7f682405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:57.689 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.682+0000 7f682affd700 1 -- 172.21.15.58:0/1530265201 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f682c077ea0 con 0x7f683c0f54d0 2021-11-20T05:37:57.689 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.682+0000 7f6839d9b700 1 --2- 172.21.15.58:0/1530265201 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f682405c300 0x7f682405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:57.690 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.682+0000 7f6839d9b700 1 --2- 172.21.15.58:0/1530265201 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f682405c300 0x7f682405e7c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f6830009740 tx=0x7f6830009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:57.690 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.686+0000 7f682affd700 1 -- 172.21.15.58:0/1530265201 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f682c04a780 con 0x7f683c0f54d0 2021-11-20T05:37:57.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: cluster 2021-11-20T05:37:56.291677+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:37:57.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: 14164) 307 : cluster [DBG] pgmap v282: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T05:37:57.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: cluster 2021-11-20T05:37:56.448350+0000 mon.smithi058 (mon.0) 789 2021-11-20T05:37:57.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: : cluster [DBG] mds.? [v2:172.21.15.187:6824/1821831102,v1:172.21.15.187:6825/1821831102] up:boot 2021-11-20T05:37:57.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: cluster 2021- 2021-11-20T05:37:57.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: 11-20T05:37:56.448414 2021-11-20T05:37:57.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: +0000 mon.smithi058 (mon.0) 790 2021-11-20T05:37:57.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: : cluster [INF] Standby daemon mds.cephfs.smithi187.fikisa assigned to filesystem cephfs as rank 0 2021-11-20T05:37:57.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:37:57.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: -11-20T05:37:56. 2021-11-20T05:37:57.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: 448639+0000 mon.smithi058 (mon.0 2021-11-20T05:37:57.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: ) 791 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby, 1 failed 2021-11-20T05:37:57.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: 2021-11-20T05:37:57.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: audit 2021 2021-11-20T05:37:57.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: -11-20T05:37:56. 2021-11-20T05:37:57.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: 448723+0000 mon.smithi058 (mon.0) 2021-11-20T05:37:57.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: 792 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.wxduhu"}]: dispatch 2021-11-20T05:37:57.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:37:57.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: -11-20T05:37: 2021-11-20T05:37:57.722 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: 56.450898+0000 mon.smithi058 (mon 2021-11-20T05:37:57.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: .0) 793 : cluster [DBG] 2021-11-20T05:37:57.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:57.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:37:57.723 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: 11-20T05:37:56. 2021-11-20T05:37:57.724 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: 532244+0000 mon.smithi058 (mon.0) 2021-11-20T05:37:57.724 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:57 smithi058 conmon[33537]: 794 : audit [DBG] from='client.? 172.21.15.58:0/3316137248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:57.878 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.881+0000 7f683b59e700 1 -- 172.21.15.58:0/1530265201 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f683c045ad0 con 0x7f683c0f54d0 2021-11-20T05:37:57.879 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.882+0000 7f682affd700 1 -- 172.21.15.58:0/1530265201 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+2571 (secure 0 0 0) 0x7f682c04dda0 con 0x7f683c0f54d0 2021-11-20T05:37:57.880 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:57.880 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":79}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:57.452063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24505,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24505":{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":80,"state":"up:resolve","state_seq":6,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:57.882 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.884+0000 7f683b59e700 1 -- 172.21.15.58:0/1530265201 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f682405c300 msgr2=0x7f682405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:57.882 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.884+0000 7f683b59e700 1 --2- 172.21.15.58:0/1530265201 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f682405c300 0x7f682405e7c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f6830009740 tx=0x7f6830009be0).stop 2021-11-20T05:37:57.883 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.884+0000 7f683b59e700 1 -- 172.21.15.58:0/1530265201 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f683c0f54d0 msgr2=0x7f683c109aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:57.883 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.884+0000 7f683b59e700 1 --2- 172.21.15.58:0/1530265201 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f683c0f54d0 0x7f683c109aa0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f682c004e30 tx=0x7f682c009b30).stop 2021-11-20T05:37:57.883 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.884+0000 7f683b59e700 1 -- 172.21.15.58:0/1530265201 shutdown_connections 2021-11-20T05:37:57.884 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.884+0000 7f683b59e700 1 --2- 172.21.15.58:0/1530265201 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f682405c300 0x7f682405e7c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:57.884 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.884+0000 7f683b59e700 1 --2- 172.21.15.58:0/1530265201 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f683c0f54d0 0x7f683c109aa0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:57.885 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.884+0000 7f683b59e700 1 --2- 172.21.15.58:0/1530265201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f683c0f6770 0x7f683c10bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:57.885 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.884+0000 7f683b59e700 1 -- 172.21.15.58:0/1530265201 >> 172.21.15.58:0/1530265201 conn(0x7f683c0f0c30 msgr2=0x7f683c0fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:57.885 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.884+0000 7f683b59e700 1 -- 172.21.15.58:0/1530265201 shutdown_connections 2021-11-20T05:37:57.886 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:57.884+0000 7f683b59e700 1 -- 172.21.15.58:0/1530265201 wait complete. 2021-11-20T05:37:57.886 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 81 2021-11-20T05:37:57.893 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:57.452063+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24505, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24505': {'gid': 24505, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 80, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.187:6827/96302868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 96302868}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 96302868}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:57.894 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24505:up:resolve 2021-11-20T05:37:57.944 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:57 smithi187 conmon[36358]: cluster 2021-11-20T05:37:56.291677+0000 mgr.smithi058.zagdlk (mgr.14164) 307 : cluster [DBG] pgmap v282: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T05:37:57.944 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:57 smithi187 conmon[36358]: cluster 2021-11-20T05:37:56.448350+0000 mon.smithi058 (mon.0) 789 : cluster [DBG] mds.? [v2:172.21.15.187:6824/1821831102,v1:172.21.15.187:6825/1821831102] up:boot 2021-11-20T05:37:57.944 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:57 smithi187 conmon[36358]: cluster 2021-11-20T05:37:56.448414+0000 mon.smithi058 (mon.0) 790 : cluster [INF] Standby daemon mds.cephfs.smithi187.fikisa assigned to filesystem cephfs as rank 0 2021-11-20T05:37:57.945 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:57 smithi187 conmon[36358]: cluster 2021-11-20T05:37:56.448639+0000 mon.smithi058 (mon.0) 791 : cluster 2021-11-20T05:37:57.945 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:57 smithi187 conmon[36358]: [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby, 1 failed 2021-11-20T05:37:57.945 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:57 smithi187 conmon[36358]: audit 2021-11-20T05:37: 2021-11-20T05:37:57.945 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:57 smithi187 conmon[36358]: 56.448723+0000 mon.smithi058 (mon.0) 792 : audit 2021-11-20T05:37:57.945 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:57 smithi187 conmon[36358]: [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.wxduhu"}]: dispatch 2021-11-20T05:37:57.946 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:57 smithi187 conmon[36358]: cluster 2021-11-20T05:37:56.450898+0000 mon.smithi058 2021-11-20T05:37:57.946 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:57 smithi187 conmon[36358]: (mon.0) 793 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:57.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:57 smithi187 conmon[36358]: audit 2021- 2021-11-20T05:37:57.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:57 smithi187 conmon[36358]: 11-20T05:37:56.532244+0000 mon.smithi058 2021-11-20T05:37:57.948 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:57 smithi187 conmon[36358]: (mon.0) 794 : audit [DBG] from='client.? 172.21.15.58:0/3316137248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:58.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:58 smithi058 conmon[33537]: cluster 2021-11-20T05:37:57.455263+0000 mon.smithi058 (mon.0 2021-11-20T05:37:58.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:58 smithi058 conmon[33537]: ) 795 : cluster [DBG] mds.? [v2:172.21.15.187:6826/96302868,v1:172.21.15.187:6827/96302868] up:resolve 2021-11-20T05:37:58.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:58 smithi058 conmon[33537]: cluster 2021-11-20T05:37:57.455369+0000 mon.smithi058 (mon.0) 796 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:58.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:58 smithi058 conmon[33537]: audit 2021-11-20T05:37:57.883974+0000 mon.smithi058 (mon.0) 797 : audit [DBG] from='client.? 172.21.15.58:0/1530265201' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:58.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:58 smithi058 conmon[33537]: cluster 2021-11-20T05:37:58.292392+0000 mgr.smithi058.zagdlk 2021-11-20T05:37:58.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:58 smithi058 conmon[33537]: (mgr.14164) 308 : cluster [DBG] pgmap v283: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2021-11-20T05:37:58.895 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:37:58.943 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:58 smithi187 conmon[36358]: cluster 2021-11-20T05:37:57.455263+0000 mon.smithi058 (mon.0) 795 : cluster [DBG] mds.? [v2:172.21.15.187:6826/96302868,v1:172.21.15.187:6827/96302868] up:resolve 2021-11-20T05:37:58.944 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:58 smithi187 conmon[36358]: 2021-11-20T05:37:58.944 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:58 smithi187 conmon[36358]: cluster 2021-11-20T05:37:57.455369+0000 mon.smithi058 (mon.0) 796 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:58.944 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:58 smithi187 conmon[36358]: audit 2021-11-20T05:37:57.883974+0000 mon.smithi058 2021-11-20T05:37:58.944 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:58 smithi187 conmon[36358]: (mon.0) 797 : audit [DBG] from='client.? 172.21.15.58:0/1530265201' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:58.945 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:58 smithi187 conmon[36358]: cluster 2021-11 2021-11-20T05:37:58.945 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:58 smithi187 conmon[36358]: -20T05:37:58.292392+0000 mgr.smithi058.zagdlk (mgr.14164) 308 : cluster [DBG] pgmap v283: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2021-11-20T05:37:59.026 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.027+0000 7f68e5782700 1 -- 172.21.15.58:0/3508095178 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f68e00f6680 msgr2=0x7f68e00f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:59.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.027+0000 7f68e5782700 1 --2- 172.21.15.58:0/3508095178 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f68e00f6680 0x7f68e00f6aa0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f68d80099e0 tx=0x7f68d8004d10).stop 2021-11-20T05:37:59.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.027+0000 7f68e5782700 1 -- 172.21.15.58:0/3508095178 shutdown_connections 2021-11-20T05:37:59.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.027+0000 7f68e5782700 1 --2- 172.21.15.58:0/3508095178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68e00f7920 0x7f68e00f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:59.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.027+0000 7f68e5782700 1 --2- 172.21.15.58:0/3508095178 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f68e00f6680 0x7f68e00f6aa0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:59.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.027+0000 7f68e5782700 1 -- 172.21.15.58:0/3508095178 >> 172.21.15.58:0/3508095178 conn(0x7f68e00f1de0 msgr2=0x7f68e00f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:59.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.027+0000 7f68e5782700 1 -- 172.21.15.58:0/3508095178 shutdown_connections 2021-11-20T05:37:59.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.027+0000 7f68e5782700 1 -- 172.21.15.58:0/3508095178 wait complete. 2021-11-20T05:37:59.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.028+0000 7f68e5782700 1 Processor -- start 2021-11-20T05:37:59.029 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.028+0000 7f68e5782700 1 -- start start 2021-11-20T05:37:59.029 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.028+0000 7f68e5782700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f68e00f6680 0x7f68e0106f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:59.029 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.028+0000 7f68e5782700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68e00f7920 0x7f68e0107480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:59.029 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.028+0000 7f68e5782700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e0107a30 con 0x7f68e00f6680 2021-11-20T05:37:59.030 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.028+0000 7f68e5782700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e0107b70 con 0x7f68e00f7920 2021-11-20T05:37:59.030 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.028+0000 7f68deffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f68e00f6680 0x7f68e0106f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:59.030 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.028+0000 7f68deffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f68e00f6680 0x7f68e0106f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41936/0 (socket says 172.21.15.58:41936) 2021-11-20T05:37:59.031 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.028+0000 7f68deffd700 1 -- 172.21.15.58:0/643974831 learned_addr learned my addr 172.21.15.58:0/643974831 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:37:59.031 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.028+0000 7f68de7fc700 1 --2- 172.21.15.58:0/643974831 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68e00f7920 0x7f68e0107480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:59.031 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.028+0000 7f68deffd700 1 -- 172.21.15.58:0/643974831 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68e00f7920 msgr2=0x7f68e0107480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:59.031 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.028+0000 7f68deffd700 1 --2- 172.21.15.58:0/643974831 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68e00f7920 0x7f68e0107480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:59.032 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.028+0000 7f68deffd700 1 -- 172.21.15.58:0/643974831 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d8009640 con 0x7f68e00f6680 2021-11-20T05:37:59.032 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.028+0000 7f68deffd700 1 --2- 172.21.15.58:0/643974831 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f68e00f6680 0x7f68e0106f90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f68d8007830 tx=0x7f68d8004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:59.032 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.029+0000 7f68d77fe700 1 -- 172.21.15.58:0/643974831 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68d800e030 con 0x7f68e00f6680 2021-11-20T05:37:59.032 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.029+0000 7f68d77fe700 1 -- 172.21.15.58:0/643974831 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68d801d8d0 con 0x7f68e00f6680 2021-11-20T05:37:59.033 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.029+0000 7f68d77fe700 1 -- 172.21.15.58:0/643974831 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68d8018070 con 0x7f68e00f6680 2021-11-20T05:37:59.033 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.029+0000 7f68e5782700 1 -- 172.21.15.58:0/643974831 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e010c5b0 con 0x7f68e00f6680 2021-11-20T05:37:59.033 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.029+0000 7f68e5782700 1 -- 172.21.15.58:0/643974831 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e010caf0 con 0x7f68e00f6680 2021-11-20T05:37:59.034 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.030+0000 7f68e5782700 1 -- 172.21.15.58:0/643974831 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e0027a00 con 0x7f68e00f6680 2021-11-20T05:37:59.035 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.030+0000 7f68d77fe700 1 -- 172.21.15.58:0/643974831 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f68d801da40 con 0x7f68e00f6680 2021-11-20T05:37:59.035 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.030+0000 7f68d77fe700 1 --2- 172.21.15.58:0/643974831 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f68cc05c2a0 0x7f68cc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:37:59.035 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.030+0000 7f68d77fe700 1 -- 172.21.15.58:0/643974831 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f68d8078270 con 0x7f68e00f6680 2021-11-20T05:37:59.035 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.031+0000 7f68de7fc700 1 --2- 172.21.15.58:0/643974831 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f68cc05c2a0 0x7f68cc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:37:59.036 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.031+0000 7f68de7fc700 1 --2- 172.21.15.58:0/643974831 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f68cc05c2a0 0x7f68cc05e760 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f68c8007fd0 tx=0x7f68c8009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:37:59.036 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.035+0000 7f68d77fe700 1 -- 172.21.15.58:0/643974831 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68d804aad0 con 0x7f68e00f6680 2021-11-20T05:37:59.228 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.230+0000 7f68e5782700 1 -- 172.21.15.58:0/643974831 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68e00fee10 con 0x7f68e00f6680 2021-11-20T05:37:59.228 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.231+0000 7f68d77fe700 1 -- 172.21.15.58:0/643974831 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2573 (secure 0 0 0) 0x7f68d804e0f0 con 0x7f68e00f6680 2021-11-20T05:37:59.229 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:37:59.229 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":79}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:37:58.453416+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24505,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24505":{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":80,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:37:59.230 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.233+0000 7f68e5782700 1 -- 172.21.15.58:0/643974831 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f68cc05c2a0 msgr2=0x7f68cc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:59.231 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.233+0000 7f68e5782700 1 --2- 172.21.15.58:0/643974831 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f68cc05c2a0 0x7f68cc05e760 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f68c8007fd0 tx=0x7f68c8009040).stop 2021-11-20T05:37:59.231 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.233+0000 7f68e5782700 1 -- 172.21.15.58:0/643974831 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f68e00f6680 msgr2=0x7f68e0106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:37:59.231 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.233+0000 7f68e5782700 1 --2- 172.21.15.58:0/643974831 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f68e00f6680 0x7f68e0106f90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f68d8007830 tx=0x7f68d8004d10).stop 2021-11-20T05:37:59.233 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.233+0000 7f68e5782700 1 -- 172.21.15.58:0/643974831 shutdown_connections 2021-11-20T05:37:59.234 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.233+0000 7f68e5782700 1 --2- 172.21.15.58:0/643974831 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f68cc05c2a0 0x7f68cc05e760 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:59.235 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.233+0000 7f68e5782700 1 --2- 172.21.15.58:0/643974831 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f68e00f6680 0x7f68e0106f90 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:59.235 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.233+0000 7f68e5782700 1 --2- 172.21.15.58:0/643974831 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68e00f7920 0x7f68e0107480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:37:59.235 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.233+0000 7f68e5782700 1 -- 172.21.15.58:0/643974831 >> 172.21.15.58:0/643974831 conn(0x7f68e00f1de0 msgr2=0x7f68e00fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:37:59.235 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.233+0000 7f68e5782700 1 -- 172.21.15.58:0/643974831 shutdown_connections 2021-11-20T05:37:59.236 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:37:59.233+0000 7f68e5782700 1 -- 172.21.15.58:0/643974831 wait complete. 2021-11-20T05:37:59.236 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 82 2021-11-20T05:37:59.242 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:37:58.453416+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24505, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24505': {'gid': 24505, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 80, 'state': 'up:reconnect', 'state_seq': 7, 'addr': '172.21.15.187:6827/96302868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 96302868}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 96302868}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:37:59.242 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24505:up:reconnect 2021-11-20T05:37:59.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:59 smithi058 conmon[33537]: cluster 2021-11-20T05:37:58.459879+0000 mon.smithi058 (mon.0) 798 : cluster [DBG] mds.? [v2:172.21.15.187:6826/96302868,v1:172.21.15.187:6827/96302868] up:reconnect 2021-11-20T05:37:59.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:59 smithi058 conmon[33537]: cluster 2021-11-20T05:37:58.459950+0000 mon.smithi058 (mon.0) 799 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:59.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:59 smithi058 conmon[33537]: cluster 2021-11-20T05 2021-11-20T05:37:59.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:59 smithi058 conmon[33537]: :37:58.480564+0000 mds.cephfs.smithi187.fikisa (mds.0) 1 : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0219997 2021-11-20T05:37:59.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:59 smithi058 conmon[33537]: 2021-11-20T05:37:59.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:59 smithi058 conmon[33537]: audit 2021-11-20T05:37:59.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:59 smithi058 conmon[33537]: 2021-11-20T05:37:59.232972+0000 mon.smithi058 2021-11-20T05:37:59.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:37:59 smithi058 conmon[33537]: (mon.0) 800 : audit [DBG] from='client.? 172.21.15.58:0/643974831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:59.944 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:59 smithi187 conmon[36358]: cluster 2021-11-20T05:37:58.459879+0000 mon.smithi058 (mon.0) 798 : cluster [DBG] 2021-11-20T05:37:59.944 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:59 smithi187 conmon[36358]: mds.? [v2:172.21.15.187:6826/96302868,v1:172.21.15.187:6827/96302868] up:reconnect 2021-11-20T05:37:59.945 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:59 smithi187 conmon[36358]: cluster 2021-11-20T05:37:58.459950+0000 mon.smithi058 (mon.0) 799 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:37:59.945 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:59 smithi187 conmon[36358]: cluster 2021-11-20T05:37:58.480564+0000 mds.cephfs.smithi187.fikisa ( 2021-11-20T05:37:59.946 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:59 smithi187 conmon[36358]: mds.0) 1 : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0219997 2021-11-20T05:37:59.946 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:59 smithi187 conmon[36358]: audit 2021-11-20T 2021-11-20T05:37:59.946 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:59 smithi187 conmon[36358]: 05:37:59.232972+0000 mon.smithi058 ( 2021-11-20T05:37:59.946 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:59 smithi187 conmon[36358]: mon.0) 800 : audit [DBG] from='client.? 172.21.15.58:0/643974831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:37:59.947 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:37:59 smithi187 conmon[36358]: 2021-11-20T05:38:00.243 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:00.387 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.387+0000 7fe863db1700 1 -- 172.21.15.58:0/2976189304 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe85c0f6840 msgr2=0x7fe85c0f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:00.387 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.387+0000 7fe863db1700 1 --2- 172.21.15.58:0/2976189304 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe85c0f6840 0x7fe85c0f6ca0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fe84c004660 tx=0x7fe84c009b30).stop 2021-11-20T05:38:00.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.387+0000 7fe863db1700 1 -- 172.21.15.58:0/2976189304 shutdown_connections 2021-11-20T05:38:00.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.387+0000 7fe863db1700 1 --2- 172.21.15.58:0/2976189304 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe85c0f6840 0x7fe85c0f6ca0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:00.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.387+0000 7fe863db1700 1 --2- 172.21.15.58:0/2976189304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe85c0f55a0 0x7fe85c0f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:38:00.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.387+0000 7fe863db1700 1 -- 172.21.15.58:0/2976189304 >> 172.21.15.58:0/2976189304 conn(0x7fe85c0f0ce0 msgr2=0x7fe85c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:00.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.387+0000 7fe863db1700 1 -- 172.21.15.58:0/2976189304 shutdown_connections 2021-11-20T05:38:00.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.387+0000 7fe863db1700 1 -- 172.21.15.58:0/2976189304 wait complete. 2021-11-20T05:38:00.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.388+0000 7fe863db1700 1 Processor -- start 2021-11-20T05:38:00.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.388+0000 7fe863db1700 1 -- start start 2021-11-20T05:38:00.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.388+0000 7fe863db1700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe85c0f55a0 0x7fe85c05ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:00.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.388+0000 7fe863db1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe85c0f6840 0x7fe85c10bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:00.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.388+0000 7fe863db1700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c10c1b0 con 0x7fe85c0f55a0 2021-11-20T05:38:00.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.388+0000 7fe863db1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c10c2f0 con 0x7fe85c0f6840 2021-11-20T05:38:00.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.388+0000 7fe861b4d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe85c0f55a0 0x7fe85c05ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:00.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.388+0000 7fe861b4d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe85c0f55a0 0x7fe85c05ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41946/0 (socket says 172.21.15.58:41946) 2021-11-20T05:38:00.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.388+0000 7fe861b4d700 1 -- 172.21.15.58:0/3116258736 learned_addr learned my addr 172.21.15.58:0/3116258736 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:00.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.388+0000 7fe86134c700 1 --2- 172.21.15.58:0/3116258736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe85c0f6840 0x7fe85c10bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:00.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.388+0000 7fe861b4d700 1 -- 172.21.15.58:0/3116258736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe85c0f6840 msgr2=0x7fe85c10bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:00.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.388+0000 7fe861b4d700 1 --2- 172.21.15.58:0/3116258736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe85c0f6840 0x7fe85c10bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:00.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.388+0000 7fe861b4d700 1 -- 172.21.15.58:0/3116258736 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84c005040 con 0x7fe85c0f55a0 2021-11-20T05:38:00.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.388+0000 7fe86134c700 1 --2- 172.21.15.58:0/3116258736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe85c0f6840 0x7fe85c10bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:38:00.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.389+0000 7fe861b4d700 1 --2- 172.21.15.58:0/3116258736 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe85c0f55a0 0x7fe85c05ff20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fe8580093f0 tx=0x7fe858006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:00.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.389+0000 7fe8527fc700 1 -- 172.21.15.58:0/3116258736 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe858009ea0 con 0x7fe85c0f55a0 2021-11-20T05:38:00.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.389+0000 7fe863db1700 1 -- 172.21.15.58:0/3116258736 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85c10c5d0 con 0x7fe85c0f55a0 2021-11-20T05:38:00.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.389+0000 7fe8527fc700 1 -- 172.21.15.58:0/3116258736 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe858013af0 con 0x7fe85c0f55a0 2021-11-20T05:38:00.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.389+0000 7fe8527fc700 1 -- 172.21.15.58:0/3116258736 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe85800dad0 con 0x7fe85c0f55a0 2021-11-20T05:38:00.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.389+0000 7fe863db1700 1 -- 172.21.15.58:0/3116258736 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe85c10cb90 con 0x7fe85c0f55a0 2021-11-20T05:38:00.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.390+0000 7fe863db1700 1 -- 172.21.15.58:0/3116258736 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe85c027a00 con 0x7fe85c0f55a0 2021-11-20T05:38:00.394 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.390+0000 7fe8527fc700 1 -- 172.21.15.58:0/3116258736 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe85801a060 con 0x7fe85c0f55a0 2021-11-20T05:38:00.395 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.391+0000 7fe8527fc700 1 --2- 172.21.15.58:0/3116258736 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe84805c2a0 0x7fe84805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:00.395 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.391+0000 7fe8527fc700 1 -- 172.21.15.58:0/3116258736 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe858076350 con 0x7fe85c0f55a0 2021-11-20T05:38:00.395 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.391+0000 7fe86134c700 1 --2- 172.21.15.58:0/3116258736 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe84805c2a0 0x7fe84805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:00.395 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.391+0000 7fe86134c700 1 --2- 172.21.15.58:0/3116258736 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe84805c2a0 0x7fe84805e760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fe84c009d60 tx=0x7fe84c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:00.396 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.395+0000 7fe8527fc700 1 -- 172.21.15.58:0/3116258736 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe858048bb0 con 0x7fe85c0f55a0 2021-11-20T05:38:00.591 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.594+0000 7fe863db1700 1 -- 172.21.15.58:0/3116258736 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe85c0fddd0 con 0x7fe85c0f55a0 2021-11-20T05:38:00.592 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.595+0000 7fe8527fc700 1 -- 172.21.15.58:0/3116258736 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2575 (secure 0 0 0) 0x7fe85804c1d0 con 0x7fe85c0f55a0 2021-11-20T05:38:00.592 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:00.593 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":84}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:00.462946+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24505,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24505":{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":80,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:00.594 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.596+0000 7fe863db1700 1 -- 172.21.15.58:0/3116258736 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe84805c2a0 msgr2=0x7fe84805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:00.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.596+0000 7fe863db1700 1 --2- 172.21.15.58:0/3116258736 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe84805c2a0 0x7fe84805e760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fe84c009d60 tx=0x7fe84c009b30).stop 2021-11-20T05:38:00.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.596+0000 7fe863db1700 1 -- 172.21.15.58:0/3116258736 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe85c0f55a0 msgr2=0x7fe85c05ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:00.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.596+0000 7fe863db1700 1 --2- 172.21.15.58:0/3116258736 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe85c0f55a0 0x7fe85c05ff20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fe8580093f0 tx=0x7fe858006ec0).stop 2021-11-20T05:38:00.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.596+0000 7fe863db1700 1 -- 172.21.15.58:0/3116258736 shutdown_connections 2021-11-20T05:38:00.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.596+0000 7fe863db1700 1 --2- 172.21.15.58:0/3116258736 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fe84805c2a0 0x7fe84805e760 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:00.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.596+0000 7fe863db1700 1 --2- 172.21.15.58:0/3116258736 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fe85c0f55a0 0x7fe85c05ff20 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:00.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.596+0000 7fe863db1700 1 --2- 172.21.15.58:0/3116258736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe85c0f6840 0x7fe85c10bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:00.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.596+0000 7fe863db1700 1 -- 172.21.15.58:0/3116258736 >> 172.21.15.58:0/3116258736 conn(0x7fe85c0f0ce0 msgr2=0x7fe85c0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:00.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.596+0000 7fe863db1700 1 -- 172.21.15.58:0/3116258736 shutdown_connections 2021-11-20T05:38:00.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:00.596+0000 7fe863db1700 1 -- 172.21.15.58:0/3116258736 wait complete. 2021-11-20T05:38:00.597 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 84 2021-11-20T05:38:00.604 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:00 smithi058 conmon[33537]: cluster 2021-11-20T05:37: 2021-11-20T05:38:00.604 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:00 smithi058 conmon[33537]: 59.460708+0000 mon.smithi058 (mon.0) 801 : cluster [DBG] mds.? [v2:172.21.15.187:6826/96302868,v1:172.21.15.187:6827/96302868] up:rejoin 2021-11-20T05:38:00.605 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:00 smithi058 conmon[33537]: cluster 2021-11-20T05:38:00.605 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:00 smithi058 conmon[33537]: 2021-11-20T05:37:59.460772+0000 mon.smithi058 (mon.0) 802 2021-11-20T05:38:00.605 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:00 smithi058 conmon[33537]: : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:00.605 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:00 smithi058 conmon[33537]: cluster 2021-11-20T05:38:00 2021-11-20T05:38:00.605 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:00 smithi058 conmon[33537]: .292900+0000 mgr.smithi058.zagdlk (mgr.14164) 309 : cluster [DBG] pgmap v284: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:00.606 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:00 smithi058 conmon[33537]: audit 2021-11- 2021-11-20T05:38:00.606 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:00 smithi058 conmon[33537]: 20T05:38:00.392907+0000 mon.smithi058 (mon.0) 803 : 2021-11-20T05:38:00.606 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:00 smithi058 conmon[33537]: audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:38:00.606 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:00 smithi058 conmon[33537]: audit 2021-11-20T05:38:00.397385+0000 mon.smithi058 (mon.0 2021-11-20T05:38:00.607 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:00 smithi058 conmon[33537]: ) 804 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:38:00.608 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:00.462946+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24505, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24505': {'gid': 24505, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 80, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.187:6827/96302868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 96302868}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 96302868}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:00.608 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24505:up:clientreplay 2021-11-20T05:38:00.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:00 smithi187 conmon[36358]: cluster 2021-11-20T05:37:59.460708+0000 mon.smithi058 (mon.0) 801 : cluster [DBG] mds.? [v2:172.21.15.187:6826/96302868,v1:172.21.15.187:6827/96302868] up:rejoin 2021-11-20T05:38:00.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:00 smithi187 conmon[36358]: cluster 2021-11-20T05:37:59.460772+0000 mon.smithi058 (mon.0) 802 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:00.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:00 smithi187 conmon[36358]: cluster 2021-11-20T05:38:00.292900 2021-11-20T05:38:00.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:00 smithi187 conmon[36358]: +0000 mgr.smithi058.zagdlk (mgr.14164) 309 : cluster [DBG] pgmap v284: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:00.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:00 smithi187 conmon[36358]: audit 2021-11-20T05:38 2021-11-20T05:38:00.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:00 smithi187 conmon[36358]: :00.392907+0000 mon.smithi058 (mon.0) 803 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:38:00.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:00 smithi187 conmon[36358]: audit 2021- 2021-11-20T05:38:00.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:00 smithi187 conmon[36358]: 11-20T05:38:00.397385+0000 mon.smithi058 2021-11-20T05:38:00.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:00 smithi187 conmon[36358]: (mon.0) 804 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:38:01.609 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:01.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:01 smithi058 conmon[33537]: cluster 2021-11-20T05:38:00.468777+0000 mon.smithi058 (mon.0 2021-11-20T05:38:01.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:01 smithi058 conmon[33537]: ) 805 : cluster [DBG] mds.? [v2:172.21.15.187:6826/96302868,v1:172.21.15.187:6827/96302868] up:clientreplay 2021-11-20T05:38:01.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:01 smithi058 conmon[33537]: cluster 2021-11-20T05:38:00.468833+0000 mon.smithi058 (mon.0) 806 : cluster [DBG] 2021-11-20T05:38:01.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:01 smithi058 conmon[33537]: mds.? [v2:172.21.15.187:6824/1821831102,v1:172.21.15.187:6825/1821831102] up:standby 2021-11-20T05:38:01.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:01 smithi058 conmon[33537]: cluster 2021- 2021-11-20T05:38:01.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:01 smithi058 conmon[33537]: 11-20T05:38:00.468898+0000 2021-11-20T05:38:01.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:01 smithi058 conmon[33537]: mon.smithi058 (mon.0) 807 : cluster [DBG] 2021-11-20T05:38:01.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:01 smithi058 conmon[33537]: fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:clientreplay,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:01.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:01 smithi058 conmon[33537]: audit 2021-11-20T05 2021-11-20T05:38:01.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:01 smithi058 conmon[33537]: :38:00.596602+0000 mon.smithi058 (mon.0 2021-11-20T05:38:01.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:01 smithi058 conmon[33537]: ) 808 : audit [DBG] from='client.? 172.21.15.58:0/3116258736' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:01.746 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.745+0000 7f821493e700 1 -- 172.21.15.58:0/2389586000 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f82100f7940 msgr2=0x7f82100f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:01.746 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.745+0000 7f821493e700 1 --2- 172.21.15.58:0/2389586000 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f82100f7940 0x7f82100f7da0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f8204004660 tx=0x7f8204009b30).stop 2021-11-20T05:38:01.747 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.745+0000 7f821493e700 1 -- 172.21.15.58:0/2389586000 shutdown_connections 2021-11-20T05:38:01.747 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.745+0000 7f821493e700 1 --2- 172.21.15.58:0/2389586000 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f82100f7940 0x7f82100f7da0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:01.747 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.745+0000 7f821493e700 1 --2- 172.21.15.58:0/2389586000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82100f66a0 0x7f82100f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:01.747 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.745+0000 7f821493e700 1 -- 172.21.15.58:0/2389586000 >> 172.21.15.58:0/2389586000 conn(0x7f82100f1de0 msgr2=0x7f82100f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:01.748 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.745+0000 7f821493e700 1 -- 172.21.15.58:0/2389586000 shutdown_connections 2021-11-20T05:38:01.748 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.745+0000 7f821493e700 1 -- 172.21.15.58:0/2389586000 wait complete. 2021-11-20T05:38:01.748 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.746+0000 7f821493e700 1 Processor -- start 2021-11-20T05:38:01.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.746+0000 7f821493e700 1 -- start start 2021-11-20T05:38:01.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.746+0000 7f821493e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82100f66a0 0x7f821006b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:01.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.746+0000 7f821493e700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f82100f7940 0x7f8210069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:01.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.746+0000 7f821493e700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821006bd70 con 0x7f82100f7940 2021-11-20T05:38:01.750 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.746+0000 7f821493e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821006a310 con 0x7f82100f66a0 2021-11-20T05:38:01.750 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.746+0000 7f820dd9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f82100f7940 0x7f8210069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:01.750 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.746+0000 7f820dd9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f82100f7940 0x7f8210069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41962/0 (socket says 172.21.15.58:41962) 2021-11-20T05:38:01.750 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.746+0000 7f820dd9b700 1 -- 172.21.15.58:0/2428980378 learned_addr learned my addr 172.21.15.58:0/2428980378 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:01.751 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.746+0000 7f820e59c700 1 --2- 172.21.15.58:0/2428980378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82100f66a0 0x7f821006b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:01.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.746+0000 7f820dd9b700 1 -- 172.21.15.58:0/2428980378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82100f66a0 msgr2=0x7f821006b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:01.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.746+0000 7f820dd9b700 1 --2- 172.21.15.58:0/2428980378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82100f66a0 0x7f821006b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:01.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.746+0000 7f820dd9b700 1 -- 172.21.15.58:0/2428980378 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8204005040 con 0x7f82100f7940 2021-11-20T05:38:01.754 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.747+0000 7f820dd9b700 1 --2- 172.21.15.58:0/2428980378 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f82100f7940 0x7f8210069e20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f8204009f50 tx=0x7f8204009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:01.754 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.747+0000 7f81feffd700 1 -- 172.21.15.58:0/2428980378 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8204018070 con 0x7f82100f7940 2021-11-20T05:38:01.754 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.747+0000 7f81feffd700 1 -- 172.21.15.58:0/2428980378 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8204013410 con 0x7f82100f7940 2021-11-20T05:38:01.755 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.747+0000 7f821493e700 1 -- 172.21.15.58:0/2428980378 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f821006a530 con 0x7f82100f7940 2021-11-20T05:38:01.755 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.747+0000 7f81feffd700 1 -- 172.21.15.58:0/2428980378 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f820401c3c0 con 0x7f82100f7940 2021-11-20T05:38:01.755 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.747+0000 7f821493e700 1 -- 172.21.15.58:0/2428980378 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821006aa90 con 0x7f82100f7940 2021-11-20T05:38:01.755 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.748+0000 7f821493e700 1 -- 172.21.15.58:0/2428980378 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8210046330 con 0x7f82100f7940 2021-11-20T05:38:01.756 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.748+0000 7f81feffd700 1 -- 172.21.15.58:0/2428980378 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f820401d420 con 0x7f82100f7940 2021-11-20T05:38:01.757 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.748+0000 7f81feffd700 1 --2- 172.21.15.58:0/2428980378 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f81f805c2a0 0x7f81f805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:01.758 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.748+0000 7f81feffd700 1 -- 172.21.15.58:0/2428980378 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8204077fa0 con 0x7f82100f7940 2021-11-20T05:38:01.758 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.749+0000 7f820e59c700 1 --2- 172.21.15.58:0/2428980378 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f81f805c2a0 0x7f81f805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:01.760 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.749+0000 7f820e59c700 1 --2- 172.21.15.58:0/2428980378 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f81f805c2a0 0x7f81f805e760 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f82100f77a0 tx=0x7f820000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:01.761 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.752+0000 7f81feffd700 1 -- 172.21.15.58:0/2428980378 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f820404a800 con 0x7f82100f7940 2021-11-20T05:38:01.943 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:01 smithi187 conmon[36358]: cluster 2021-11-20T05:38:00.468777+0000 mon.smithi058 (mon.0) 805 : cluster [DBG] 2021-11-20T05:38:01.944 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:01 smithi187 conmon[36358]: mds.? [v2:172.21.15.187:6826/96302868,v1:172.21.15.187:6827/96302868] up:clientreplay 2021-11-20T05:38:01.944 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:01 smithi187 conmon[36358]: cluster 2021-11-20T05:38:00.468833+0000 mon.smithi058 (mon.0) 806 : cluster [DBG] mds.? [v2:172.21.15.187:6824/1821831102,v1:172.21.15.187:6825/1821831102] up:standby 2021-11-20T05:38:01.944 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:01 smithi187 conmon[36358]: cluster 2021-11-20T 2021-11-20T05:38:01.944 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:01 smithi187 conmon[36358]: 05:38:00.468898+0000 mon.smithi058 (mon.0) 807 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:clientreplay,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:01.945 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:01 smithi187 conmon[36358]: audit 2021-11-20T05:38: 2021-11-20T05:38:01.945 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:01 smithi187 conmon[36358]: 00.596602+0000 mon.smithi058 (mon.0) 808 : audit 2021-11-20T05:38:01.945 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:01 smithi187 conmon[36358]: [DBG] from='client.? 172.21.15.58:0/3116258736' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:01.946 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.949+0000 7f821493e700 1 -- 172.21.15.58:0/2428980378 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8210027a00 con 0x7f82100f7940 2021-11-20T05:38:01.947 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.950+0000 7f81feffd700 1 -- 172.21.15.58:0/2428980378 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f820404de20 con 0x7f82100f7940 2021-11-20T05:38:01.947 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:01.947 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":84}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:00.462946+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24505,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24505":{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":80,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:01.949 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.951+0000 7f821493e700 1 -- 172.21.15.58:0/2428980378 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f81f805c2a0 msgr2=0x7f81f805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:01.949 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.951+0000 7f821493e700 1 --2- 172.21.15.58:0/2428980378 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f81f805c2a0 0x7f81f805e760 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f82100f77a0 tx=0x7f820000b040).stop 2021-11-20T05:38:01.949 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.951+0000 7f821493e700 1 -- 172.21.15.58:0/2428980378 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f82100f7940 msgr2=0x7f8210069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:01.950 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.951+0000 7f821493e700 1 --2- 172.21.15.58:0/2428980378 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f82100f7940 0x7f8210069e20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f8204009f50 tx=0x7f8204009b30).stop 2021-11-20T05:38:01.950 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.951+0000 7f821493e700 1 -- 172.21.15.58:0/2428980378 shutdown_connections 2021-11-20T05:38:01.950 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.951+0000 7f821493e700 1 --2- 172.21.15.58:0/2428980378 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f81f805c2a0 0x7f81f805e760 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:01.950 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.951+0000 7f821493e700 1 --2- 172.21.15.58:0/2428980378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82100f66a0 0x7f821006b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:01.950 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.951+0000 7f821493e700 1 --2- 172.21.15.58:0/2428980378 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f82100f7940 0x7f8210069e20 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:01.951 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.951+0000 7f821493e700 1 -- 172.21.15.58:0/2428980378 >> 172.21.15.58:0/2428980378 conn(0x7f82100f1de0 msgr2=0x7f82100fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:01.952 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.951+0000 7f821493e700 1 -- 172.21.15.58:0/2428980378 shutdown_connections 2021-11-20T05:38:01.952 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:01.951+0000 7f821493e700 1 -- 172.21.15.58:0/2428980378 wait complete. 2021-11-20T05:38:01.952 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 84 2021-11-20T05:38:01.961 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:00.462946+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24505, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24505': {'gid': 24505, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 80, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.187:6827/96302868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 96302868}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 96302868}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:01.962 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24505:up:clientreplay 2021-11-20T05:38:02.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:02 smithi058 conmon[33537]: audit 2021-11-20T05:38:01.951942+0000 mon.smithi058 (mon.0) 809 : audit [DBG] from='client.? 172.21.15.58:0/2428980378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:02.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:02 smithi058 conmon[33537]: 2021-11-20T05:38:02.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:02 smithi058 conmon[33537]: cluster 2021-11-20T05:38:02.293551+0000 mgr.smithi058.zagdlk (mgr.14164) 310 : cluster [DBG] pgmap v285: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:02.944 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:02 smithi187 conmon[36358]: audit 2021-11-20T05:38:01.951942+0000 mon.smithi058 (mon.0) 809 : audit [DBG] from='client.? 172.21.15.58:0/2428980378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:02.944 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:02 smithi187 conmon[36358]: cluster 2021-11-20T05:38:02.293551+0000 mgr.smithi058.zagdlk (mgr.14164) 310 : cluster [DBG] pgmap v285: 65 pgs: 65 active+clean; 3.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:02.963 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:03.097 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.098+0000 7f8b32f34700 1 -- 172.21.15.58:0/3800493073 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b2c0f7940 msgr2=0x7f8b2c0f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:03.097 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.098+0000 7f8b32f34700 1 --2- 172.21.15.58:0/3800493073 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b2c0f7940 0x7f8b2c0f7da0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f8b18004660 tx=0x7f8b18009b30).stop 2021-11-20T05:38:03.098 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.098+0000 7f8b32f34700 1 -- 172.21.15.58:0/3800493073 shutdown_connections 2021-11-20T05:38:03.098 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.098+0000 7f8b32f34700 1 --2- 172.21.15.58:0/3800493073 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b2c0f7940 0x7f8b2c0f7da0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:03.098 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.098+0000 7f8b32f34700 1 --2- 172.21.15.58:0/3800493073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b2c0f66a0 0x7f8b2c0f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:03.098 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.098+0000 7f8b32f34700 1 -- 172.21.15.58:0/3800493073 >> 172.21.15.58:0/3800493073 conn(0x7f8b2c0f1da0 msgr2=0x7f8b2c0f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:03.099 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.098+0000 7f8b32f34700 1 -- 172.21.15.58:0/3800493073 shutdown_connections 2021-11-20T05:38:03.099 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.098+0000 7f8b32f34700 1 -- 172.21.15.58:0/3800493073 wait complete. 2021-11-20T05:38:03.099 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.099+0000 7f8b32f34700 1 Processor -- start 2021-11-20T05:38:03.099 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.099+0000 7f8b32f34700 1 -- start start 2021-11-20T05:38:03.100 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.099+0000 7f8b32f34700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b2c0f66a0 0x7f8b2c106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:03.100 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.099+0000 7f8b32f34700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b2c0f7940 0x7f8b2c1073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:03.100 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.099+0000 7f8b32f34700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c107960 con 0x7f8b2c0f7940 2021-11-20T05:38:03.100 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.099+0000 7f8b32f34700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c107aa0 con 0x7f8b2c0f66a0 2021-11-20T05:38:03.100 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.099+0000 7f8b2bfff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b2c0f7940 0x7f8b2c1073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:03.101 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.099+0000 7f8b2bfff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b2c0f7940 0x7f8b2c1073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41974/0 (socket says 172.21.15.58:41974) 2021-11-20T05:38:03.101 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.099+0000 7f8b2bfff700 1 -- 172.21.15.58:0/3987647507 learned_addr learned my addr 172.21.15.58:0/3987647507 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:03.103 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.099+0000 7f8b30cd0700 1 --2- 172.21.15.58:0/3987647507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b2c0f66a0 0x7f8b2c106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:03.103 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.099+0000 7f8b2bfff700 1 -- 172.21.15.58:0/3987647507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b2c0f66a0 msgr2=0x7f8b2c106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:03.103 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.099+0000 7f8b2bfff700 1 --2- 172.21.15.58:0/3987647507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b2c0f66a0 0x7f8b2c106ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:03.103 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.099+0000 7f8b2bfff700 1 -- 172.21.15.58:0/3987647507 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b18005040 con 0x7f8b2c0f7940 2021-11-20T05:38:03.103 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.100+0000 7f8b2bfff700 1 --2- 172.21.15.58:0/3987647507 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b2c0f7940 0x7f8b2c1073b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f8b18009830 tx=0x7f8b18009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:03.104 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.100+0000 7f8b297fa700 1 -- 172.21.15.58:0/3987647507 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b18018070 con 0x7f8b2c0f7940 2021-11-20T05:38:03.104 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.100+0000 7f8b297fa700 1 -- 172.21.15.58:0/3987647507 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b18013410 con 0x7f8b2c0f7940 2021-11-20T05:38:03.104 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.100+0000 7f8b32f34700 1 -- 172.21.15.58:0/3987647507 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b2c10c4e0 con 0x7f8b2c0f7940 2021-11-20T05:38:03.104 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.100+0000 7f8b297fa700 1 -- 172.21.15.58:0/3987647507 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b1801c620 con 0x7f8b2c0f7940 2021-11-20T05:38:03.105 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.100+0000 7f8b32f34700 1 -- 172.21.15.58:0/3987647507 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b2c10caa0 con 0x7f8b2c0f7940 2021-11-20T05:38:03.106 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.101+0000 7f8b32f34700 1 -- 172.21.15.58:0/3987647507 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b2c027a00 con 0x7f8b2c0f7940 2021-11-20T05:38:03.106 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.101+0000 7f8b297fa700 1 -- 172.21.15.58:0/3987647507 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8b1801c8e0 con 0x7f8b2c0f7940 2021-11-20T05:38:03.107 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.101+0000 7f8b297fa700 1 --2- 172.21.15.58:0/3987647507 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b1c05c2a0 0x7f8b1c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:03.107 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.101+0000 7f8b297fa700 1 -- 172.21.15.58:0/3987647507 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8b18078200 con 0x7f8b2c0f7940 2021-11-20T05:38:03.107 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.102+0000 7f8b30cd0700 1 --2- 172.21.15.58:0/3987647507 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b1c05c2a0 0x7f8b1c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:03.107 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.102+0000 7f8b30cd0700 1 --2- 172.21.15.58:0/3987647507 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b1c05c2a0 0x7f8b1c05e760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f8b20007fd0 tx=0x7f8b20009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:03.108 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.105+0000 7f8b297fa700 1 -- 172.21.15.58:0/3987647507 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b1804aa00 con 0x7f8b2c0f7940 2021-11-20T05:38:03.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.323+0000 7f8b32f34700 1 -- 172.21.15.58:0/3987647507 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b2c0fee30 con 0x7f8b2c0f7940 2021-11-20T05:38:03.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.323+0000 7f8b297fa700 1 -- 172.21.15.58:0/3987647507 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f8b1804e020 con 0x7f8b2c0f7940 2021-11-20T05:38:03.321 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:03.321 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":84}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:00.462946+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24505,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24505":{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":80,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:03.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.325+0000 7f8b32f34700 1 -- 172.21.15.58:0/3987647507 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b1c05c2a0 msgr2=0x7f8b1c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:03.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.325+0000 7f8b32f34700 1 --2- 172.21.15.58:0/3987647507 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b1c05c2a0 0x7f8b1c05e760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f8b20007fd0 tx=0x7f8b20009040).stop 2021-11-20T05:38:03.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.325+0000 7f8b32f34700 1 -- 172.21.15.58:0/3987647507 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b2c0f7940 msgr2=0x7f8b2c1073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:03.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.325+0000 7f8b32f34700 1 --2- 172.21.15.58:0/3987647507 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b2c0f7940 0x7f8b2c1073b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f8b18009830 tx=0x7f8b18009b30).stop 2021-11-20T05:38:03.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.325+0000 7f8b32f34700 1 -- 172.21.15.58:0/3987647507 shutdown_connections 2021-11-20T05:38:03.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.325+0000 7f8b32f34700 1 --2- 172.21.15.58:0/3987647507 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b1c05c2a0 0x7f8b1c05e760 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:03.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.325+0000 7f8b32f34700 1 --2- 172.21.15.58:0/3987647507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b2c0f66a0 0x7f8b2c106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:03.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.325+0000 7f8b32f34700 1 --2- 172.21.15.58:0/3987647507 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b2c0f7940 0x7f8b2c1073b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:03.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.325+0000 7f8b32f34700 1 -- 172.21.15.58:0/3987647507 >> 172.21.15.58:0/3987647507 conn(0x7f8b2c0f1da0 msgr2=0x7f8b2c0fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:03.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.325+0000 7f8b32f34700 1 -- 172.21.15.58:0/3987647507 shutdown_connections 2021-11-20T05:38:03.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:03.325+0000 7f8b32f34700 1 -- 172.21.15.58:0/3987647507 wait complete. 2021-11-20T05:38:03.326 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 84 2021-11-20T05:38:03.335 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:00.462946+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24505, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24505': {'gid': 24505, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 80, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.187:6827/96302868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 96302868}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 96302868}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:03.336 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24505:up:clientreplay 2021-11-20T05:38:03.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:03 smithi058 conmon[33537]: audit 2021-11-20T05:38:03.325182+0000 2021-11-20T05:38:03.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:03 smithi058 conmon[33537]: mon.smithi058 (mon.0) 810 : audit [DBG] from='client.? 172.21.15.58:0/3987647507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:03.943 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:03 smithi187 conmon[36358]: audit 2021-11-20T05:38:03.325182+0000 mon.smithi058 (mon.0) 810 : audit [DBG] from='client.? 172.21.15.58:0/3987647507' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:04.337 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:04.481 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.482+0000 7f2ab56db700 1 -- 172.21.15.58:0/364770711 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2ab00f6680 msgr2=0x7f2ab00f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:04.482 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.482+0000 7f2ab56db700 1 --2- 172.21.15.58:0/364770711 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2ab00f6680 0x7f2ab00f6aa0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f2aa0004660 tx=0x7f2aa0009b30).stop 2021-11-20T05:38:04.482 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.482+0000 7f2ab56db700 1 -- 172.21.15.58:0/364770711 shutdown_connections 2021-11-20T05:38:04.482 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.482+0000 7f2ab56db700 1 --2- 172.21.15.58:0/364770711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab00f7920 0x7f2ab00f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:04.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.482+0000 7f2ab56db700 1 --2- 172.21.15.58:0/364770711 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2ab00f6680 0x7f2ab00f6aa0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:04.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.482+0000 7f2ab56db700 1 -- 172.21.15.58:0/364770711 >> 172.21.15.58:0/364770711 conn(0x7f2ab00f1de0 msgr2=0x7f2ab00f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:04.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.482+0000 7f2ab56db700 1 -- 172.21.15.58:0/364770711 shutdown_connections 2021-11-20T05:38:04.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.482+0000 7f2ab56db700 1 -- 172.21.15.58:0/364770711 wait complete. 2021-11-20T05:38:04.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.483+0000 7f2ab56db700 1 Processor -- start 2021-11-20T05:38:04.484 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.483+0000 7f2ab56db700 1 -- start start 2021-11-20T05:38:04.484 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.483+0000 7f2ab56db700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab00f6680 0x7f2ab006b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:04.484 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.483+0000 7f2ab56db700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2ab00f7920 0x7f2ab0069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:04.484 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.483+0000 7f2ab56db700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab006bd70 con 0x7f2ab00f7920 2021-11-20T05:38:04.485 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.483+0000 7f2ab56db700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab006a310 con 0x7f2ab00f6680 2021-11-20T05:38:04.485 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.484+0000 7f2aae7fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2ab00f7920 0x7f2ab0069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:04.485 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.484+0000 7f2aae7fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2ab00f7920 0x7f2ab0069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41986/0 (socket says 172.21.15.58:41986) 2021-11-20T05:38:04.485 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.484+0000 7f2aae7fc700 1 -- 172.21.15.58:0/1695647008 learned_addr learned my addr 172.21.15.58:0/1695647008 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:04.486 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.484+0000 7f2aaeffd700 1 --2- 172.21.15.58:0/1695647008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab00f6680 0x7f2ab006b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:04.486 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.484+0000 7f2aae7fc700 1 -- 172.21.15.58:0/1695647008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab00f6680 msgr2=0x7f2ab006b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:04.486 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.484+0000 7f2aae7fc700 1 --2- 172.21.15.58:0/1695647008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab00f6680 0x7f2ab006b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:04.486 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.484+0000 7f2aae7fc700 1 -- 172.21.15.58:0/1695647008 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa0005040 con 0x7f2ab00f7920 2021-11-20T05:38:04.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.484+0000 7f2aae7fc700 1 --2- 172.21.15.58:0/1695647008 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2ab00f7920 0x7f2ab0069e20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f2aa40026d0 tx=0x7f2aa4009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:04.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.484+0000 7f2a977fe700 1 -- 172.21.15.58:0/1695647008 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2aa40079c0 con 0x7f2ab00f7920 2021-11-20T05:38:04.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.484+0000 7f2a977fe700 1 -- 172.21.15.58:0/1695647008 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2aa400f040 con 0x7f2ab00f7920 2021-11-20T05:38:04.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.484+0000 7f2ab56db700 1 -- 172.21.15.58:0/1695647008 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab006a590 con 0x7f2ab00f7920 2021-11-20T05:38:04.487 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.484+0000 7f2a977fe700 1 -- 172.21.15.58:0/1695647008 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2aa40148b0 con 0x7f2ab00f7920 2021-11-20T05:38:04.488 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.484+0000 7f2ab56db700 1 -- 172.21.15.58:0/1695647008 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab006ab50 con 0x7f2ab00f7920 2021-11-20T05:38:04.489 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.485+0000 7f2ab56db700 1 -- 172.21.15.58:0/1695647008 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab0027a00 con 0x7f2ab00f7920 2021-11-20T05:38:04.489 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.485+0000 7f2a977fe700 1 -- 172.21.15.58:0/1695647008 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2aa4007b20 con 0x7f2ab00f7920 2021-11-20T05:38:04.489 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.486+0000 7f2a977fe700 1 --2- 172.21.15.58:0/1695647008 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2a9805c2a0 0x7f2a9805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:04.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.486+0000 7f2a977fe700 1 -- 172.21.15.58:0/1695647008 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2aa4076220 con 0x7f2ab00f7920 2021-11-20T05:38:04.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.486+0000 7f2aaeffd700 1 --2- 172.21.15.58:0/1695647008 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2a9805c2a0 0x7f2a9805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:04.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.486+0000 7f2aaeffd700 1 --2- 172.21.15.58:0/1695647008 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2a9805c2a0 0x7f2a9805e760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f2aa0009f50 tx=0x7f2aa0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:04.490 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.490+0000 7f2a977fe700 1 -- 172.21.15.58:0/1695647008 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2aa4048b00 con 0x7f2ab00f7920 2021-11-20T05:38:04.684 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.688+0000 7f2ab56db700 1 -- 172.21.15.58:0/1695647008 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ab00fee10 con 0x7f2ab00f7920 2021-11-20T05:38:04.685 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.688+0000 7f2a977fe700 1 -- 172.21.15.58:0/1695647008 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f2aa404c120 con 0x7f2ab00f7920 2021-11-20T05:38:04.686 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:04.686 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":84}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:00.462946+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24505,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24505":{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":80,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:04.687 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.690+0000 7f2ab56db700 1 -- 172.21.15.58:0/1695647008 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2a9805c2a0 msgr2=0x7f2a9805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:04.688 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.690+0000 7f2ab56db700 1 --2- 172.21.15.58:0/1695647008 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2a9805c2a0 0x7f2a9805e760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f2aa0009f50 tx=0x7f2aa0009b30).stop 2021-11-20T05:38:04.688 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.690+0000 7f2ab56db700 1 -- 172.21.15.58:0/1695647008 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2ab00f7920 msgr2=0x7f2ab0069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:04.688 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.690+0000 7f2ab56db700 1 --2- 172.21.15.58:0/1695647008 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2ab00f7920 0x7f2ab0069e20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f2aa40026d0 tx=0x7f2aa4009de0).stop 2021-11-20T05:38:04.688 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.690+0000 7f2ab56db700 1 -- 172.21.15.58:0/1695647008 shutdown_connections 2021-11-20T05:38:04.689 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.690+0000 7f2ab56db700 1 --2- 172.21.15.58:0/1695647008 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f2a9805c2a0 0x7f2a9805e760 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:04.689 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.690+0000 7f2ab56db700 1 --2- 172.21.15.58:0/1695647008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ab00f6680 0x7f2ab006b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:04.689 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.690+0000 7f2ab56db700 1 --2- 172.21.15.58:0/1695647008 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f2ab00f7920 0x7f2ab0069e20 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:04.689 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.690+0000 7f2ab56db700 1 -- 172.21.15.58:0/1695647008 >> 172.21.15.58:0/1695647008 conn(0x7f2ab00f1de0 msgr2=0x7f2ab00fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:04.689 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.690+0000 7f2ab56db700 1 -- 172.21.15.58:0/1695647008 shutdown_connections 2021-11-20T05:38:04.690 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:04.690+0000 7f2ab56db700 1 -- 172.21.15.58:0/1695647008 wait complete. 2021-11-20T05:38:04.690 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 84 2021-11-20T05:38:04.699 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:00.462946+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24505, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24505': {'gid': 24505, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 80, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.187:6827/96302868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 96302868}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 96302868}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:04.699 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24505:up:clientreplay 2021-11-20T05:38:05.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:05 smithi187 conmon[36358]: cluster 2021-11-20T05:38:04.294220+0000 mgr.smithi058.zagdlk (mgr.14164) 311 : cluster [DBG] pgmap v286: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T05:38:05.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:05 smithi187 conmon[36358]: audit 2021-11-20T05:38:04.690121+0000 2021-11-20T05:38:05.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:05 smithi187 conmon[36358]: mon.smithi058 (mon.0) 811 : audit [DBG] from='client.? 172.21.15.58:0/1695647008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:05.700 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:05.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:05 smithi058 conmon[33537]: cluster 2021-11-20T05:38:04.294220+0000 mgr.smithi058.zagdlk (mgr.14164) 311 : 2021-11-20T05:38:05.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:05 smithi058 conmon[33537]: cluster [DBG] pgmap v286: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 1 op/s 2021-11-20T05:38:05.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:05 smithi058 conmon[33537]: audit 2021-11-20T05:38:04.690121+0000 mon.smithi058 (mon.0) 811 : audit [DBG] from='client.? 172.21.15.58:0/1695647008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:05.832 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.835+0000 7f6545a94700 1 -- 172.21.15.58:0/1065306799 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65400f5fc0 msgr2=0x7f65400f6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:05.834 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.835+0000 7f6545a94700 1 --2- 172.21.15.58:0/1065306799 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65400f5fc0 0x7f65400f6420 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f6530004660 tx=0x7f6530009b30).stop 2021-11-20T05:38:05.834 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.835+0000 7f6545a94700 1 -- 172.21.15.58:0/1065306799 shutdown_connections 2021-11-20T05:38:05.835 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.835+0000 7f6545a94700 1 --2- 172.21.15.58:0/1065306799 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65400f5fc0 0x7f65400f6420 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:05.835 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.835+0000 7f6545a94700 1 --2- 172.21.15.58:0/1065306799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65400f4d20 0x7f65400f5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:05.835 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.835+0000 7f6545a94700 1 -- 172.21.15.58:0/1065306799 >> 172.21.15.58:0/1065306799 conn(0x7f65400f0440 msgr2=0x7f65400f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:05.835 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.835+0000 7f6545a94700 1 -- 172.21.15.58:0/1065306799 shutdown_connections 2021-11-20T05:38:05.836 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.835+0000 7f6545a94700 1 -- 172.21.15.58:0/1065306799 wait complete. 2021-11-20T05:38:05.836 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.836+0000 7f6545a94700 1 Processor -- start 2021-11-20T05:38:05.838 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.836+0000 7f6545a94700 1 -- start start 2021-11-20T05:38:05.839 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.836+0000 7f6545a94700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65400f4d20 0x7f6540105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:05.839 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.836+0000 7f6545a94700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65400f5fc0 0x7f6540106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:05.839 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.836+0000 7f6545a94700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6540106890 con 0x7f65400f4d20 2021-11-20T05:38:05.840 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.836+0000 7f6545a94700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65401069d0 con 0x7f65400f5fc0 2021-11-20T05:38:05.840 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.836+0000 7f653f7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65400f4d20 0x7f6540105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:05.840 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.836+0000 7f653f7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65400f4d20 0x7f6540105da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:41996/0 (socket says 172.21.15.58:41996) 2021-11-20T05:38:05.840 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.836+0000 7f653f7fe700 1 -- 172.21.15.58:0/589300944 learned_addr learned my addr 172.21.15.58:0/589300944 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:05.841 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.837+0000 7f653effd700 1 --2- 172.21.15.58:0/589300944 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65400f5fc0 0x7f6540106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:05.841 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.837+0000 7f653f7fe700 1 -- 172.21.15.58:0/589300944 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65400f5fc0 msgr2=0x7f6540106290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:05.841 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.837+0000 7f653f7fe700 1 --2- 172.21.15.58:0/589300944 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65400f5fc0 0x7f6540106290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:05.841 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.837+0000 7f653f7fe700 1 -- 172.21.15.58:0/589300944 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6530005040 con 0x7f65400f4d20 2021-11-20T05:38:05.842 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.837+0000 7f653f7fe700 1 --2- 172.21.15.58:0/589300944 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65400f4d20 0x7f6540105da0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f6528002700 tx=0x7f6528009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:05.842 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.837+0000 7f6537fff700 1 -- 172.21.15.58:0/589300944 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6528009320 con 0x7f65400f4d20 2021-11-20T05:38:05.842 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.837+0000 7f6545a94700 1 -- 172.21.15.58:0/589300944 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f654010b3e0 con 0x7f65400f4d20 2021-11-20T05:38:05.842 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.837+0000 7f6537fff700 1 -- 172.21.15.58:0/589300944 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f652800f040 con 0x7f65400f4d20 2021-11-20T05:38:05.843 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.837+0000 7f6537fff700 1 -- 172.21.15.58:0/589300944 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6528014900 con 0x7f65400f4d20 2021-11-20T05:38:05.843 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.837+0000 7f6545a94700 1 -- 172.21.15.58:0/589300944 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f654010b9a0 con 0x7f65400f4d20 2021-11-20T05:38:05.843 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.838+0000 7f6545a94700 1 -- 172.21.15.58:0/589300944 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65400fd4b0 con 0x7f65400f4d20 2021-11-20T05:38:05.844 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.838+0000 7f6537fff700 1 -- 172.21.15.58:0/589300944 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6528009480 con 0x7f65400f4d20 2021-11-20T05:38:05.844 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.839+0000 7f6537fff700 1 --2- 172.21.15.58:0/589300944 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f652c05c2a0 0x7f652c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:05.844 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.839+0000 7f6537fff700 1 -- 172.21.15.58:0/589300944 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6528076660 con 0x7f65400f4d20 2021-11-20T05:38:05.845 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.839+0000 7f653effd700 1 --2- 172.21.15.58:0/589300944 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f652c05c2a0 0x7f652c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:05.845 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.839+0000 7f653effd700 1 --2- 172.21.15.58:0/589300944 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f652c05c2a0 0x7f652c05e760 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f6530004e30 tx=0x7f6530009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:05.846 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:05.844+0000 7f6537fff700 1 -- 172.21.15.58:0/589300944 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6528048ec0 con 0x7f65400f4d20 2021-11-20T05:38:06.035 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:06.038+0000 7f6545a94700 1 -- 172.21.15.58:0/589300944 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f654010c1d0 con 0x7f65400f4d20 2021-11-20T05:38:06.036 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:06.039+0000 7f6537fff700 1 -- 172.21.15.58:0/589300944 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f652804c4e0 con 0x7f65400f4d20 2021-11-20T05:38:06.037 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:06.037 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":84}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:00.462946+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24505,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24505":{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":80,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:06.038 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:06.040+0000 7f6545a94700 1 -- 172.21.15.58:0/589300944 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f652c05c2a0 msgr2=0x7f652c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:06.039 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:06.040+0000 7f6545a94700 1 --2- 172.21.15.58:0/589300944 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f652c05c2a0 0x7f652c05e760 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f6530004e30 tx=0x7f6530009b30).stop 2021-11-20T05:38:06.039 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:06.040+0000 7f6545a94700 1 -- 172.21.15.58:0/589300944 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65400f4d20 msgr2=0x7f6540105da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:06.039 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:06.040+0000 7f6545a94700 1 --2- 172.21.15.58:0/589300944 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65400f4d20 0x7f6540105da0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f6528002700 tx=0x7f6528009d80).stop 2021-11-20T05:38:06.039 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:06.040+0000 7f6545a94700 1 -- 172.21.15.58:0/589300944 shutdown_connections 2021-11-20T05:38:06.040 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:06.040+0000 7f6545a94700 1 --2- 172.21.15.58:0/589300944 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f652c05c2a0 0x7f652c05e760 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:06.040 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:06.040+0000 7f6545a94700 1 --2- 172.21.15.58:0/589300944 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f65400f4d20 0x7f6540105da0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:06.040 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:06.040+0000 7f6545a94700 1 --2- 172.21.15.58:0/589300944 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65400f5fc0 0x7f6540106290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:06.040 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:06.040+0000 7f6545a94700 1 -- 172.21.15.58:0/589300944 >> 172.21.15.58:0/589300944 conn(0x7f65400f0440 msgr2=0x7f65400fa300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:06.041 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:06.041+0000 7f6545a94700 1 -- 172.21.15.58:0/589300944 shutdown_connections 2021-11-20T05:38:06.041 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:06.041+0000 7f6545a94700 1 -- 172.21.15.58:0/589300944 wait complete. 2021-11-20T05:38:06.041 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 84 2021-11-20T05:38:06.049 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:00.462946+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24505, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24505': {'gid': 24505, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 80, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.187:6827/96302868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 96302868}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 96302868}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:06.050 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24505:up:clientreplay 2021-11-20T05:38:06.465 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:06 smithi058 conmon[33537]: audit 2021-11-20T05:38:06.040871+0000 mon.smithi058 2021-11-20T05:38:06.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:06 smithi058 conmon[33537]: (mon.0) 812 : audit [DBG] from='client.? 172.21.15.58:0/589300944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:06.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:06 smithi187 conmon[36358]: audit 2021-11-20T05:38:06.040871+0000 mon.smithi058 (mon.0) 812 : audit [DBG] from='client.? 172.21.15.58:0/589300944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:07.052 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:07.187 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.188+0000 7f5a1cf15700 1 -- 172.21.15.58:0/552068031 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5a180f66c0 msgr2=0x7f5a180f6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:07.187 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.188+0000 7f5a1cf15700 1 --2- 172.21.15.58:0/552068031 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5a180f66c0 0x7f5a180f6ae0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f5a100099e0 tx=0x7f5a10004d10).stop 2021-11-20T05:38:07.187 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.188+0000 7f5a1cf15700 1 -- 172.21.15.58:0/552068031 shutdown_connections 2021-11-20T05:38:07.188 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.188+0000 7f5a1cf15700 1 --2- 172.21.15.58:0/552068031 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a180f7960 0x7f5a180f7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:07.188 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.188+0000 7f5a1cf15700 1 --2- 172.21.15.58:0/552068031 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5a180f66c0 0x7f5a180f6ae0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:07.188 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.188+0000 7f5a1cf15700 1 -- 172.21.15.58:0/552068031 >> 172.21.15.58:0/552068031 conn(0x7f5a180f1e40 msgr2=0x7f5a180f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:07.188 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.188+0000 7f5a1cf15700 1 -- 172.21.15.58:0/552068031 shutdown_connections 2021-11-20T05:38:07.189 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.188+0000 7f5a1cf15700 1 -- 172.21.15.58:0/552068031 wait complete. 2021-11-20T05:38:07.189 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.188+0000 7f5a1cf15700 1 Processor -- start 2021-11-20T05:38:07.189 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.189+0000 7f5a1cf15700 1 -- start start 2021-11-20T05:38:07.189 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.189+0000 7f5a1cf15700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a180f66c0 0x7f5a1806b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:07.190 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.189+0000 7f5a1cf15700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5a180f7960 0x7f5a18069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:07.190 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.189+0000 7f5a1cf15700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1806bd70 con 0x7f5a180f7960 2021-11-20T05:38:07.190 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.189+0000 7f5a1cf15700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1806a310 con 0x7f5a180f66c0 2021-11-20T05:38:07.190 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.189+0000 7f5a15d9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5a180f7960 0x7f5a18069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:07.191 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.189+0000 7f5a15d9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5a180f7960 0x7f5a18069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42008/0 (socket says 172.21.15.58:42008) 2021-11-20T05:38:07.191 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.189+0000 7f5a15d9b700 1 -- 172.21.15.58:0/2750291920 learned_addr learned my addr 172.21.15.58:0/2750291920 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:07.191 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.189+0000 7f5a1659c700 1 --2- 172.21.15.58:0/2750291920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a180f66c0 0x7f5a1806b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:07.191 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.189+0000 7f5a15d9b700 1 -- 172.21.15.58:0/2750291920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a180f66c0 msgr2=0x7f5a1806b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:07.191 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.189+0000 7f5a15d9b700 1 --2- 172.21.15.58:0/2750291920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a180f66c0 0x7f5a1806b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:07.192 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.189+0000 7f5a15d9b700 1 -- 172.21.15.58:0/2750291920 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a10009640 con 0x7f5a180f7960 2021-11-20T05:38:07.192 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.189+0000 7f5a1659c700 1 --2- 172.21.15.58:0/2750291920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a180f66c0 0x7f5a1806b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:38:07.192 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.189+0000 7f5a15d9b700 1 --2- 172.21.15.58:0/2750291920 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5a180f7960 0x7f5a18069e20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f5a000026d0 tx=0x7f5a00009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:07.192 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.189+0000 7f5a0effd700 1 -- 172.21.15.58:0/2750291920 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a00009320 con 0x7f5a180f7960 2021-11-20T05:38:07.193 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.189+0000 7f5a0effd700 1 -- 172.21.15.58:0/2750291920 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a0000f040 con 0x7f5a180f7960 2021-11-20T05:38:07.193 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.189+0000 7f5a1cf15700 1 -- 172.21.15.58:0/2750291920 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1806a590 con 0x7f5a180f7960 2021-11-20T05:38:07.193 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.190+0000 7f5a0effd700 1 -- 172.21.15.58:0/2750291920 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a000147e0 con 0x7f5a180f7960 2021-11-20T05:38:07.194 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.190+0000 7f5a1cf15700 1 -- 172.21.15.58:0/2750291920 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a1806aad0 con 0x7f5a180f7960 2021-11-20T05:38:07.194 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.190+0000 7f5a1cf15700 1 -- 172.21.15.58:0/2750291920 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a18046330 con 0x7f5a180f7960 2021-11-20T05:38:07.194 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.191+0000 7f5a0effd700 1 -- 172.21.15.58:0/2750291920 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5a00009480 con 0x7f5a180f7960 2021-11-20T05:38:07.195 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.192+0000 7f5a0effd700 1 --2- 172.21.15.58:0/2750291920 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5a0405c2a0 0x7f5a0405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:07.195 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.192+0000 7f5a0effd700 1 -- 172.21.15.58:0/2750291920 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5a00076db0 con 0x7f5a180f7960 2021-11-20T05:38:07.196 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.192+0000 7f5a1659c700 1 --2- 172.21.15.58:0/2750291920 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5a0405c2a0 0x7f5a0405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:07.196 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.192+0000 7f5a1659c700 1 --2- 172.21.15.58:0/2750291920 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5a0405c2a0 0x7f5a0405e760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f5a1000b3b0 tx=0x7f5a10004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:07.196 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.195+0000 7f5a0effd700 1 -- 172.21.15.58:0/2750291920 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a000495d0 con 0x7f5a180f7960 2021-11-20T05:38:07.387 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.392+0000 7f5a1cf15700 1 -- 172.21.15.58:0/2750291920 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a180fee50 con 0x7f5a180f7960 2021-11-20T05:38:07.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.392+0000 7f5a0effd700 1 -- 172.21.15.58:0/2750291920 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f5a0004cbf0 con 0x7f5a180f7960 2021-11-20T05:38:07.389 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:07.389 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":84}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:00.462946+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24505,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24505":{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":80,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:07.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.394+0000 7f5a1cf15700 1 -- 172.21.15.58:0/2750291920 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5a0405c2a0 msgr2=0x7f5a0405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:07.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.394+0000 7f5a1cf15700 1 --2- 172.21.15.58:0/2750291920 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5a0405c2a0 0x7f5a0405e760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f5a1000b3b0 tx=0x7f5a10004d10).stop 2021-11-20T05:38:07.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.394+0000 7f5a1cf15700 1 -- 172.21.15.58:0/2750291920 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5a180f7960 msgr2=0x7f5a18069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:07.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.394+0000 7f5a1cf15700 1 --2- 172.21.15.58:0/2750291920 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5a180f7960 0x7f5a18069e20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f5a000026d0 tx=0x7f5a00009de0).stop 2021-11-20T05:38:07.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.394+0000 7f5a1cf15700 1 -- 172.21.15.58:0/2750291920 shutdown_connections 2021-11-20T05:38:07.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.394+0000 7f5a1cf15700 1 --2- 172.21.15.58:0/2750291920 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5a0405c2a0 0x7f5a0405e760 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:07.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.394+0000 7f5a1cf15700 1 --2- 172.21.15.58:0/2750291920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5a180f66c0 0x7f5a1806b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:07.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.394+0000 7f5a1cf15700 1 --2- 172.21.15.58:0/2750291920 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5a180f7960 0x7f5a18069e20 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:07.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.394+0000 7f5a1cf15700 1 -- 172.21.15.58:0/2750291920 >> 172.21.15.58:0/2750291920 conn(0x7f5a180f1e40 msgr2=0x7f5a180fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:07.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.394+0000 7f5a1cf15700 1 -- 172.21.15.58:0/2750291920 shutdown_connections 2021-11-20T05:38:07.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:07.394+0000 7f5a1cf15700 1 -- 172.21.15.58:0/2750291920 wait complete. 2021-11-20T05:38:07.393 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 84 2021-11-20T05:38:07.400 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:07 smithi058 conmon[33537]: cluster 2021-11-20T05:38:06.294754+0000 2021-11-20T05:38:07.401 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:07 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 312 : cluster [DBG] pgmap v287: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 202 B/s wr, 1 op/s 2021-11-20T05:38:07.402 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:00.462946+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24505, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24505': {'gid': 24505, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 80, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.187:6827/96302868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 96302868}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 96302868}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:07.402 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24505:up:clientreplay 2021-11-20T05:38:07.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:07 smithi187 conmon[36358]: cluster 2021-11-20T05:38:06.294754+0000 mgr.smithi058.zagdlk (mgr.14164) 312 : cluster [DBG] pgmap v287: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 202 B/s wr, 1 op/s 2021-11-20T05:38:08.404 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:08.541 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.542+0000 7efbf07dd700 1 -- 172.21.15.58:0/3961877619 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbe80f6820 msgr2=0x7efbe80f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:08.542 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.542+0000 7efbf07dd700 1 --2- 172.21.15.58:0/3961877619 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbe80f6820 0x7efbe80f6c80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7efbdc004660 tx=0x7efbdc009b30).stop 2021-11-20T05:38:08.542 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.542+0000 7efbf07dd700 1 -- 172.21.15.58:0/3961877619 shutdown_connections 2021-11-20T05:38:08.542 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.542+0000 7efbf07dd700 1 --2- 172.21.15.58:0/3961877619 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbe80f6820 0x7efbe80f6c80 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:08.542 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.542+0000 7efbf07dd700 1 --2- 172.21.15.58:0/3961877619 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbe80f5580 0x7efbe80f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:08.542 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.542+0000 7efbf07dd700 1 -- 172.21.15.58:0/3961877619 >> 172.21.15.58:0/3961877619 conn(0x7efbe80f0ce0 msgr2=0x7efbe80f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:08.543 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.542+0000 7efbf07dd700 1 -- 172.21.15.58:0/3961877619 shutdown_connections 2021-11-20T05:38:08.543 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.542+0000 7efbf07dd700 1 -- 172.21.15.58:0/3961877619 wait complete. 2021-11-20T05:38:08.543 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.542+0000 7efbf07dd700 1 Processor -- start 2021-11-20T05:38:08.543 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.542+0000 7efbf07dd700 1 -- start start 2021-11-20T05:38:08.544 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.543+0000 7efbf07dd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbe80f5580 0x7efbe8105dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:08.544 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.543+0000 7efbf07dd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbe80f6820 0x7efbe81062b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:08.544 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.543+0000 7efbf07dd700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbe8106860 con 0x7efbe80f6820 2021-11-20T05:38:08.544 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.543+0000 7efbf07dd700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbe81069a0 con 0x7efbe80f5580 2021-11-20T05:38:08.545 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.543+0000 7efbedd78700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbe80f6820 0x7efbe81062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:08.545 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.543+0000 7efbedd78700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbe80f6820 0x7efbe81062b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42018/0 (socket says 172.21.15.58:42018) 2021-11-20T05:38:08.545 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.543+0000 7efbedd78700 1 -- 172.21.15.58:0/3676708787 learned_addr learned my addr 172.21.15.58:0/3676708787 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:08.545 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.543+0000 7efbee579700 1 --2- 172.21.15.58:0/3676708787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbe80f5580 0x7efbe8105dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:08.546 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.543+0000 7efbedd78700 1 -- 172.21.15.58:0/3676708787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbe80f5580 msgr2=0x7efbe8105dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:08.546 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.543+0000 7efbedd78700 1 --2- 172.21.15.58:0/3676708787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbe80f5580 0x7efbe8105dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:08.546 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.543+0000 7efbedd78700 1 -- 172.21.15.58:0/3676708787 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbdc005040 con 0x7efbe80f6820 2021-11-20T05:38:08.546 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.543+0000 7efbedd78700 1 --2- 172.21.15.58:0/3676708787 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbe80f6820 0x7efbe81062b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7efbdc009f50 tx=0x7efbdc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:08.546 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.543+0000 7efbe6ffd700 1 -- 172.21.15.58:0/3676708787 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efbdc018070 con 0x7efbe80f6820 2021-11-20T05:38:08.547 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.543+0000 7efbe6ffd700 1 -- 172.21.15.58:0/3676708787 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efbdc013410 con 0x7efbe80f6820 2021-11-20T05:38:08.547 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.543+0000 7efbf07dd700 1 -- 172.21.15.58:0/3676708787 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbe810b3e0 con 0x7efbe80f6820 2021-11-20T05:38:08.547 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.543+0000 7efbe6ffd700 1 -- 172.21.15.58:0/3676708787 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efbdc01c3c0 con 0x7efbe80f6820 2021-11-20T05:38:08.547 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.543+0000 7efbf07dd700 1 -- 172.21.15.58:0/3676708787 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbe810b9a0 con 0x7efbe80f6820 2021-11-20T05:38:08.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.544+0000 7efbf07dd700 1 -- 172.21.15.58:0/3676708787 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbe8027a00 con 0x7efbe80f6820 2021-11-20T05:38:08.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.545+0000 7efbe6ffd700 1 -- 172.21.15.58:0/3676708787 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7efbdc01d420 con 0x7efbe80f6820 2021-11-20T05:38:08.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.545+0000 7efbe6ffd700 1 --2- 172.21.15.58:0/3676708787 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efbd005c2a0 0x7efbd005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:08.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.545+0000 7efbe6ffd700 1 -- 172.21.15.58:0/3676708787 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efbdc077fa0 con 0x7efbe80f6820 2021-11-20T05:38:08.548 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.545+0000 7efbee579700 1 --2- 172.21.15.58:0/3676708787 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efbd005c2a0 0x7efbd005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:08.549 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.546+0000 7efbee579700 1 --2- 172.21.15.58:0/3676708787 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efbd005c2a0 0x7efbd005e760 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7efbe80f6680 tx=0x7efbe0009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:08.549 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.549+0000 7efbe6ffd700 1 -- 172.21.15.58:0/3676708787 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efbdc04a800 con 0x7efbe80f6820 2021-11-20T05:38:08.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:08 smithi187 conmon[36358]: audit 2021-11-20T05:38:07.393284+0000 mon.smithi058 (mon.0 2021-11-20T05:38:08.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:08 smithi187 conmon[36358]: ) 813 : audit [DBG] from='client.? 172.21.15.58:0/2750291920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:08.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:08 smithi058 conmon[33537]: audit 2021-11-20T05:38:07.393284+0000 mon.smithi058 (mon.0) 813 : audit [DBG] from='client.? 172.21.15.58:0/2750291920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:08.745 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.749+0000 7efbf07dd700 1 -- 172.21.15.58:0/3676708787 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efbe80fdd10 con 0x7efbe80f6820 2021-11-20T05:38:08.745 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.749+0000 7efbe6ffd700 1 -- 172.21.15.58:0/3676708787 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2575 (secure 0 0 0) 0x7efbdc04de20 con 0x7efbe80f6820 2021-11-20T05:38:08.746 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:08.746 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":84}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:00.462946+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24505,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24505":{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":80,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:08.747 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.751+0000 7efbf07dd700 1 -- 172.21.15.58:0/3676708787 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efbd005c2a0 msgr2=0x7efbd005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:08.747 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.751+0000 7efbf07dd700 1 --2- 172.21.15.58:0/3676708787 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efbd005c2a0 0x7efbd005e760 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7efbe80f6680 tx=0x7efbe0009040).stop 2021-11-20T05:38:08.748 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.751+0000 7efbf07dd700 1 -- 172.21.15.58:0/3676708787 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbe80f6820 msgr2=0x7efbe81062b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:08.748 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.751+0000 7efbf07dd700 1 --2- 172.21.15.58:0/3676708787 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbe80f6820 0x7efbe81062b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7efbdc009f50 tx=0x7efbdc009b30).stop 2021-11-20T05:38:08.748 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.751+0000 7efbf07dd700 1 -- 172.21.15.58:0/3676708787 shutdown_connections 2021-11-20T05:38:08.748 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.751+0000 7efbf07dd700 1 --2- 172.21.15.58:0/3676708787 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efbd005c2a0 0x7efbd005e760 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:08.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.751+0000 7efbf07dd700 1 --2- 172.21.15.58:0/3676708787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbe80f5580 0x7efbe8105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:08.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.751+0000 7efbf07dd700 1 --2- 172.21.15.58:0/3676708787 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efbe80f6820 0x7efbe81062b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:08.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.751+0000 7efbf07dd700 1 -- 172.21.15.58:0/3676708787 >> 172.21.15.58:0/3676708787 conn(0x7efbe80f0ce0 msgr2=0x7efbe80fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:08.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.751+0000 7efbf07dd700 1 -- 172.21.15.58:0/3676708787 shutdown_connections 2021-11-20T05:38:08.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:08.751+0000 7efbf07dd700 1 -- 172.21.15.58:0/3676708787 wait complete. 2021-11-20T05:38:08.750 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 84 2021-11-20T05:38:08.759 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:00.462946+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24505, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24505': {'gid': 24505, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 80, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.187:6827/96302868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 96302868}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 96302868}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:08.760 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24505:up:clientreplay 2021-11-20T05:38:09.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:09 smithi187 conmon[36358]: cluster 2021-11-20T05:38:08.295270+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:38:09.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:09 smithi187 conmon[36358]: 313 : cluster [DBG] pgmap v288: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T05:38:09.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:09 smithi187 conmon[36358]: audit 2021-11-20T05:38:08.750496+0000 mon.smithi058 (mon.0) 814 : audit [DBG] from='client.? 172.21.15.58:0/3676708787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:09.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:09 smithi058 conmon[33537]: cluster 2021-11-20T05:38:08.295270+0000 mgr.smithi058.zagdlk (mgr.14164) 313 : cluster [DBG] pgmap v288: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 170 B/s wr, 1 op/s 2021-11-20T05:38:09.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:09 smithi058 conmon[33537]: audit 2021-11-20T05:38:08.750496+0000 mon.smithi058 (mon.0) 814 : audit [DBG] from='client.? 172.21.15.58:0/3676708787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:09.760 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:09.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.896+0000 7ff263ca6700 1 -- 172.21.15.58:0/398931227 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff25c0f7d90 msgr2=0x7ff25c0f81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:09.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.896+0000 7ff263ca6700 1 --2- 172.21.15.58:0/398931227 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff25c0f7d90 0x7ff25c0f81b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7ff24c004660 tx=0x7ff24c009b30).stop 2021-11-20T05:38:09.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.896+0000 7ff263ca6700 1 -- 172.21.15.58:0/398931227 shutdown_connections 2021-11-20T05:38:09.896 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.896+0000 7ff263ca6700 1 --2- 172.21.15.58:0/398931227 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff25c0f4780 0x7ff25c0f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:09.896 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.896+0000 7ff263ca6700 1 --2- 172.21.15.58:0/398931227 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff25c0f7d90 0x7ff25c0f81b0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:09.896 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.896+0000 7ff263ca6700 1 -- 172.21.15.58:0/398931227 >> 172.21.15.58:0/398931227 conn(0x7ff25c0f04a0 msgr2=0x7ff25c0f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:09.896 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.896+0000 7ff263ca6700 1 -- 172.21.15.58:0/398931227 shutdown_connections 2021-11-20T05:38:09.896 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.896+0000 7ff263ca6700 1 -- 172.21.15.58:0/398931227 wait complete. 2021-11-20T05:38:09.897 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.897+0000 7ff263ca6700 1 Processor -- start 2021-11-20T05:38:09.897 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.897+0000 7ff263ca6700 1 -- start start 2021-11-20T05:38:09.897 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.897+0000 7ff263ca6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff25c0f4780 0x7ff25c0f7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:09.897 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.897+0000 7ff263ca6700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff25c0f7d90 0x7ff25c0f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:09.898 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.897+0000 7ff263ca6700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0f7900 con 0x7ff25c0f7d90 2021-11-20T05:38:09.898 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.897+0000 7ff263ca6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0f5ea0 con 0x7ff25c0f4780 2021-11-20T05:38:09.898 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.897+0000 7ff261241700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff25c0f7d90 0x7ff25c0f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:09.898 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.897+0000 7ff261241700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff25c0f7d90 0x7ff25c0f59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42030/0 (socket says 172.21.15.58:42030) 2021-11-20T05:38:09.898 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.897+0000 7ff261241700 1 -- 172.21.15.58:0/3720559298 learned_addr learned my addr 172.21.15.58:0/3720559298 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:09.899 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.897+0000 7ff261a42700 1 --2- 172.21.15.58:0/3720559298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff25c0f4780 0x7ff25c0f7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:09.899 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.897+0000 7ff261241700 1 -- 172.21.15.58:0/3720559298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff25c0f4780 msgr2=0x7ff25c0f7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:09.899 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.897+0000 7ff261241700 1 --2- 172.21.15.58:0/3720559298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff25c0f4780 0x7ff25c0f7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:09.899 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.897+0000 7ff261241700 1 -- 172.21.15.58:0/3720559298 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c005040 con 0x7ff25c0f7d90 2021-11-20T05:38:09.900 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.897+0000 7ff261a42700 1 --2- 172.21.15.58:0/3720559298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff25c0f4780 0x7ff25c0f7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:38:09.900 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.897+0000 7ff261241700 1 --2- 172.21.15.58:0/3720559298 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff25c0f7d90 0x7ff25c0f59b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7ff25800c790 tx=0x7ff258007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:09.900 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.898+0000 7ff2527fc700 1 -- 172.21.15.58:0/3720559298 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff25800e680 con 0x7ff25c0f7d90 2021-11-20T05:38:09.900 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.898+0000 7ff263ca6700 1 -- 172.21.15.58:0/3720559298 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff25c0f6120 con 0x7ff25c0f7d90 2021-11-20T05:38:09.900 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.898+0000 7ff2527fc700 1 -- 172.21.15.58:0/3720559298 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff25800ec90 con 0x7ff25c0f7d90 2021-11-20T05:38:09.901 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.898+0000 7ff2527fc700 1 -- 172.21.15.58:0/3720559298 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff258012cb0 con 0x7ff25c0f7d90 2021-11-20T05:38:09.901 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.898+0000 7ff263ca6700 1 -- 172.21.15.58:0/3720559298 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff25c0f66e0 con 0x7ff25c0f7d90 2021-11-20T05:38:09.901 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.899+0000 7ff263ca6700 1 -- 172.21.15.58:0/3720559298 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff25c027a00 con 0x7ff25c0f7d90 2021-11-20T05:38:09.902 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.899+0000 7ff2527fc700 1 -- 172.21.15.58:0/3720559298 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff258010070 con 0x7ff25c0f7d90 2021-11-20T05:38:09.903 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.899+0000 7ff2527fc700 1 --2- 172.21.15.58:0/3720559298 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff24805c2a0 0x7ff24805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:09.903 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.900+0000 7ff2527fc700 1 -- 172.21.15.58:0/3720559298 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff258076410 con 0x7ff25c0f7d90 2021-11-20T05:38:09.903 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.900+0000 7ff261a42700 1 --2- 172.21.15.58:0/3720559298 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff24805c2a0 0x7ff24805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:09.903 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.900+0000 7ff261a42700 1 --2- 172.21.15.58:0/3720559298 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff24805c2a0 0x7ff24805e760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7ff24c009d60 tx=0x7ff24c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:09.904 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:09.904+0000 7ff2527fc700 1 -- 172.21.15.58:0/3720559298 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff258048c70 con 0x7ff25c0f7d90 2021-11-20T05:38:10.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:10.100+0000 7ff263ca6700 1 -- 172.21.15.58:0/3720559298 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff25c0fef40 con 0x7ff25c0f7d90 2021-11-20T05:38:10.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:10.100+0000 7ff2527fc700 1 -- 172.21.15.58:0/3720559298 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2575 (secure 0 0 0) 0x7ff25804c290 con 0x7ff25c0f7d90 2021-11-20T05:38:10.096 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:10.097 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":84}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:00.462946+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24505,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24505":{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":80,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:10.098 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:10.102+0000 7ff263ca6700 1 -- 172.21.15.58:0/3720559298 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff24805c2a0 msgr2=0x7ff24805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:10.098 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:10.102+0000 7ff263ca6700 1 --2- 172.21.15.58:0/3720559298 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff24805c2a0 0x7ff24805e760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7ff24c009d60 tx=0x7ff24c009b30).stop 2021-11-20T05:38:10.099 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:10.102+0000 7ff263ca6700 1 -- 172.21.15.58:0/3720559298 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff25c0f7d90 msgr2=0x7ff25c0f59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:10.099 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:10.102+0000 7ff263ca6700 1 --2- 172.21.15.58:0/3720559298 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff25c0f7d90 0x7ff25c0f59b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7ff25800c790 tx=0x7ff258007d90).stop 2021-11-20T05:38:10.099 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:10.102+0000 7ff263ca6700 1 -- 172.21.15.58:0/3720559298 shutdown_connections 2021-11-20T05:38:10.099 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:10.102+0000 7ff263ca6700 1 --2- 172.21.15.58:0/3720559298 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7ff24805c2a0 0x7ff24805e760 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:10.100 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:10.102+0000 7ff263ca6700 1 --2- 172.21.15.58:0/3720559298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff25c0f4780 0x7ff25c0f7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:10.100 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:10.102+0000 7ff263ca6700 1 --2- 172.21.15.58:0/3720559298 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7ff25c0f7d90 0x7ff25c0f59b0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:10.100 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:10.102+0000 7ff263ca6700 1 -- 172.21.15.58:0/3720559298 >> 172.21.15.58:0/3720559298 conn(0x7ff25c0f04a0 msgr2=0x7ff25c0fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:10.100 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:10.102+0000 7ff263ca6700 1 -- 172.21.15.58:0/3720559298 shutdown_connections 2021-11-20T05:38:10.100 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:10.102+0000 7ff263ca6700 1 -- 172.21.15.58:0/3720559298 wait complete. 2021-11-20T05:38:10.101 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 84 2021-11-20T05:38:10.110 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:00.462946+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24505, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24505': {'gid': 24505, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 80, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.187:6827/96302868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 96302868}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 96302868}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:10.110 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24505:up:clientreplay 2021-11-20T05:38:10.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:10 smithi058 conmon[33537]: audit 2021-11-20T05:38:10.101135+0000 mon.smithi058 (mon.0) 815 : audit [DBG] from='client.? 172.21.15.58:0/3720559298' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:10.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:10 smithi187 conmon[36358]: audit 2021-11-20T05:38:10.101135+0000 mon.smithi058 (mon.0) 815 : audit [DBG] from='client.? 172.21.15.58:0/3720559298' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:11.112 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:11.247 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.248+0000 7f86e8961700 1 -- 172.21.15.58:0/3604747303 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f86e40f6770 msgr2=0x7f86e40f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:11.247 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.248+0000 7f86e8961700 1 --2- 172.21.15.58:0/3604747303 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f86e40f6770 0x7f86e40f6bd0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f86d8004660 tx=0x7f86d8009b30).stop 2021-11-20T05:38:11.248 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.249+0000 7f86e8961700 1 -- 172.21.15.58:0/3604747303 shutdown_connections 2021-11-20T05:38:11.249 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.249+0000 7f86e8961700 1 --2- 172.21.15.58:0/3604747303 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f86e40f6770 0x7f86e40f6bd0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:11.249 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.249+0000 7f86e8961700 1 --2- 172.21.15.58:0/3604747303 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86e40f54d0 0x7f86e40f58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:11.249 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.249+0000 7f86e8961700 1 -- 172.21.15.58:0/3604747303 >> 172.21.15.58:0/3604747303 conn(0x7f86e40f0c10 msgr2=0x7f86e40f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:11.250 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.249+0000 7f86e8961700 1 -- 172.21.15.58:0/3604747303 shutdown_connections 2021-11-20T05:38:11.250 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.249+0000 7f86e8961700 1 -- 172.21.15.58:0/3604747303 wait complete. 2021-11-20T05:38:11.250 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.249+0000 7f86e8961700 1 Processor -- start 2021-11-20T05:38:11.250 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.249+0000 7f86e8961700 1 -- start start 2021-11-20T05:38:11.250 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.250+0000 7f86e8961700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f86e40f54d0 0x7f86e4106d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:11.251 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.250+0000 7f86e8961700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86e40f6770 0x7f86e4107240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:11.251 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.250+0000 7f86e8961700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e41077f0 con 0x7f86e40f54d0 2021-11-20T05:38:11.251 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.250+0000 7f86e8961700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e4107ff0 con 0x7f86e40f6770 2021-11-20T05:38:11.251 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.250+0000 7f86e37fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f86e40f54d0 0x7f86e4106d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:11.252 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.250+0000 7f86e37fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f86e40f54d0 0x7f86e4106d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42038/0 (socket says 172.21.15.58:42038) 2021-11-20T05:38:11.252 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.250+0000 7f86e37fe700 1 -- 172.21.15.58:0/3681151382 learned_addr learned my addr 172.21.15.58:0/3681151382 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:11.252 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.250+0000 7f86e2ffd700 1 --2- 172.21.15.58:0/3681151382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86e40f6770 0x7f86e4107240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:11.252 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.250+0000 7f86e37fe700 1 -- 172.21.15.58:0/3681151382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86e40f6770 msgr2=0x7f86e4107240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:11.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.250+0000 7f86e37fe700 1 --2- 172.21.15.58:0/3681151382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86e40f6770 0x7f86e4107240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:11.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.250+0000 7f86e37fe700 1 -- 172.21.15.58:0/3681151382 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d8005040 con 0x7f86e40f54d0 2021-11-20T05:38:11.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.250+0000 7f86e37fe700 1 --2- 172.21.15.58:0/3681151382 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f86e40f54d0 0x7f86e4106d50 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f86d400e7c0 tx=0x7f86d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:11.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.250+0000 7f86cbfff700 1 -- 172.21.15.58:0/3681151382 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86d40092e0 con 0x7f86e40f54d0 2021-11-20T05:38:11.253 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.250+0000 7f86cbfff700 1 -- 172.21.15.58:0/3681151382 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86d400f040 con 0x7f86e40f54d0 2021-11-20T05:38:11.254 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.250+0000 7f86cbfff700 1 -- 172.21.15.58:0/3681151382 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86d40147c0 con 0x7f86e40f54d0 2021-11-20T05:38:11.254 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.250+0000 7f86e8961700 1 -- 172.21.15.58:0/3681151382 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e41082d0 con 0x7f86e40f54d0 2021-11-20T05:38:11.254 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.250+0000 7f86e8961700 1 -- 172.21.15.58:0/3681151382 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e4108890 con 0x7f86e40f54d0 2021-11-20T05:38:11.254 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.251+0000 7f86e8961700 1 -- 172.21.15.58:0/3681151382 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e40f48c0 con 0x7f86e40f54d0 2021-11-20T05:38:11.256 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.252+0000 7f86cbfff700 1 -- 172.21.15.58:0/3681151382 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f86d401b030 con 0x7f86e40f54d0 2021-11-20T05:38:11.256 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.252+0000 7f86cbfff700 1 --2- 172.21.15.58:0/3681151382 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f86cc05c2f0 0x7f86cc05e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:11.256 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.252+0000 7f86cbfff700 1 -- 172.21.15.58:0/3681151382 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f86d4076230 con 0x7f86e40f54d0 2021-11-20T05:38:11.256 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.252+0000 7f86e2ffd700 1 --2- 172.21.15.58:0/3681151382 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f86cc05c2f0 0x7f86cc05e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:11.257 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.253+0000 7f86e2ffd700 1 --2- 172.21.15.58:0/3681151382 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f86cc05c2f0 0x7f86cc05e7b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f86d8004d80 tx=0x7f86d8016040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:11.257 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.256+0000 7f86cbfff700 1 -- 172.21.15.58:0/3681151382 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86d4048a90 con 0x7f86e40f54d0 2021-11-20T05:38:11.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.453+0000 7f86e8961700 1 -- 172.21.15.58:0/3681151382 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86e4045ad0 con 0x7f86e40f54d0 2021-11-20T05:38:11.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.453+0000 7f86cbfff700 1 -- 172.21.15.58:0/3681151382 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f86d4019020 con 0x7f86e40f54d0 2021-11-20T05:38:11.450 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:11.450 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":84}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:00.462946+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24505,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24505":{"gid":24505,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":80,"state":"up:clientreplay","state_seq":9,"addr":"172.21.15.187:6827/96302868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":96302868},{"type":"v1","addr":"172.21.15.187:6827","nonce":96302868}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:11.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.455+0000 7f86e8961700 1 -- 172.21.15.58:0/3681151382 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f86cc05c2f0 msgr2=0x7f86cc05e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:11.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.455+0000 7f86e8961700 1 --2- 172.21.15.58:0/3681151382 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f86cc05c2f0 0x7f86cc05e7b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f86d8004d80 tx=0x7f86d8016040).stop 2021-11-20T05:38:11.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.455+0000 7f86e8961700 1 -- 172.21.15.58:0/3681151382 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f86e40f54d0 msgr2=0x7f86e4106d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:11.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.455+0000 7f86e8961700 1 --2- 172.21.15.58:0/3681151382 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f86e40f54d0 0x7f86e4106d50 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f86d400e7c0 tx=0x7f86d400b040).stop 2021-11-20T05:38:11.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.455+0000 7f86e8961700 1 -- 172.21.15.58:0/3681151382 shutdown_connections 2021-11-20T05:38:11.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.455+0000 7f86e8961700 1 --2- 172.21.15.58:0/3681151382 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f86cc05c2f0 0x7f86cc05e7b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:11.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.455+0000 7f86e8961700 1 --2- 172.21.15.58:0/3681151382 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f86e40f54d0 0x7f86e4106d50 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:11.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.455+0000 7f86e8961700 1 --2- 172.21.15.58:0/3681151382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86e40f6770 0x7f86e4107240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:11.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.455+0000 7f86e8961700 1 -- 172.21.15.58:0/3681151382 >> 172.21.15.58:0/3681151382 conn(0x7f86e40f0c10 msgr2=0x7f86e405ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:11.454 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.455+0000 7f86e8961700 1 -- 172.21.15.58:0/3681151382 shutdown_connections 2021-11-20T05:38:11.454 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:11.455+0000 7f86e8961700 1 -- 172.21.15.58:0/3681151382 wait complete. 2021-11-20T05:38:11.454 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 84 2021-11-20T05:38:11.461 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:11 smithi058 conmon[33537]: cluster 2021-11-20T05:38:10.295709+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:38:11.462 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:11 smithi058 conmon[33537]: mgr.14164) 314 : cluster [DBG] pgmap v289: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T05:38:11.463 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:00.462946+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24505, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24505': {'gid': 24505, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 80, 'state': 'up:clientreplay', 'state_seq': 9, 'addr': '172.21.15.187:6827/96302868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 96302868}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 96302868}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:11.464 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24505:up:clientreplay 2021-11-20T05:38:11.570 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:11 smithi187 conmon[36358]: cluster 2021-11-20T05:38:10.295709+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:38:11.570 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:11 smithi187 conmon[36358]: 14164) 314 : cluster [DBG] pgmap v289: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 170 B/s wr, 0 op/s 2021-11-20T05:38:12.465 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:12.601 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.601+0000 7fadd733e700 1 -- 172.21.15.58:0/803427954 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fadd00f54d0 msgr2=0x7fadd00f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:12.601 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.601+0000 7fadd733e700 1 --2- 172.21.15.58:0/803427954 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fadd00f54d0 0x7fadd00f58f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fadcc004660 tx=0x7fadcc009b30).stop 2021-11-20T05:38:12.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.602+0000 7fadd733e700 1 -- 172.21.15.58:0/803427954 shutdown_connections 2021-11-20T05:38:12.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.602+0000 7fadd733e700 1 --2- 172.21.15.58:0/803427954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadd00f6770 0x7fadd00f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:12.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.602+0000 7fadd733e700 1 --2- 172.21.15.58:0/803427954 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fadd00f54d0 0x7fadd00f58f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:12.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.602+0000 7fadd733e700 1 -- 172.21.15.58:0/803427954 >> 172.21.15.58:0/803427954 conn(0x7fadd00f0c30 msgr2=0x7fadd00f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:12.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.602+0000 7fadd733e700 1 -- 172.21.15.58:0/803427954 shutdown_connections 2021-11-20T05:38:12.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.602+0000 7fadd733e700 1 -- 172.21.15.58:0/803427954 wait complete. 2021-11-20T05:38:12.604 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.602+0000 7fadd733e700 1 Processor -- start 2021-11-20T05:38:12.604 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.602+0000 7fadd733e700 1 -- start start 2021-11-20T05:38:12.604 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.602+0000 7fadd733e700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fadd00f54d0 0x7fadd0109aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:12.604 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.602+0000 7fadd733e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadd00f6770 0x7fadd010bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:12.605 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.602+0000 7fadd733e700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd0060520 con 0x7fadd00f54d0 2021-11-20T05:38:12.605 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.602+0000 7fadd733e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd0060660 con 0x7fadd00f6770 2021-11-20T05:38:12.605 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.602+0000 7fadd633c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fadd00f54d0 0x7fadd0109aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:12.605 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.602+0000 7fadd633c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fadd00f54d0 0x7fadd0109aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42048/0 (socket says 172.21.15.58:42048) 2021-11-20T05:38:12.606 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.602+0000 7fadd633c700 1 -- 172.21.15.58:0/616458844 learned_addr learned my addr 172.21.15.58:0/616458844 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:12.606 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.603+0000 7fadd5b3b700 1 --2- 172.21.15.58:0/616458844 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadd00f6770 0x7fadd010bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:12.606 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.603+0000 7fadd633c700 1 -- 172.21.15.58:0/616458844 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadd00f6770 msgr2=0x7fadd010bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:12.606 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.603+0000 7fadd633c700 1 --2- 172.21.15.58:0/616458844 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadd00f6770 0x7fadd010bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:12.607 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.603+0000 7fadd633c700 1 -- 172.21.15.58:0/616458844 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc005040 con 0x7fadd00f54d0 2021-11-20T05:38:12.607 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.603+0000 7fadd5b3b700 1 --2- 172.21.15.58:0/616458844 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadd00f6770 0x7fadd010bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:38:12.607 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.603+0000 7fadd633c700 1 --2- 172.21.15.58:0/616458844 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fadd00f54d0 0x7fadd0109aa0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fadcc004e30 tx=0x7fadcc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:12.607 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.603+0000 7fadc6ffd700 1 -- 172.21.15.58:0/616458844 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadcc018070 con 0x7fadd00f54d0 2021-11-20T05:38:12.608 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.603+0000 7fadd733e700 1 -- 172.21.15.58:0/616458844 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd010c4f0 con 0x7fadd00f54d0 2021-11-20T05:38:12.608 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.603+0000 7fadc6ffd700 1 -- 172.21.15.58:0/616458844 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fadcc013410 con 0x7fadd00f54d0 2021-11-20T05:38:12.608 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.603+0000 7fadc6ffd700 1 -- 172.21.15.58:0/616458844 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadcc01c3c0 con 0x7fadd00f54d0 2021-11-20T05:38:12.608 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.603+0000 7fadd733e700 1 -- 172.21.15.58:0/616458844 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd010ca20 con 0x7fadd00f54d0 2021-11-20T05:38:12.608 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.604+0000 7fadd733e700 1 -- 172.21.15.58:0/616458844 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd00f48c0 con 0x7fadd00f54d0 2021-11-20T05:38:12.609 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.604+0000 7fadc6ffd700 1 -- 172.21.15.58:0/616458844 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fadcc01d420 con 0x7fadd00f54d0 2021-11-20T05:38:12.609 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.605+0000 7fadc6ffd700 1 --2- 172.21.15.58:0/616458844 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fadbc05c300 0x7fadbc05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:12.609 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.605+0000 7fadc6ffd700 1 -- 172.21.15.58:0/616458844 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fadcc077fc0 con 0x7fadd00f54d0 2021-11-20T05:38:12.609 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.605+0000 7fadd5b3b700 1 --2- 172.21.15.58:0/616458844 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fadbc05c300 0x7fadbc05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:12.610 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.605+0000 7fadd5b3b700 1 --2- 172.21.15.58:0/616458844 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fadbc05c300 0x7fadbc05e7c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fadc0009740 tx=0x7fadc0009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:12.610 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.609+0000 7fadc6ffd700 1 -- 172.21.15.58:0/616458844 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fadcc04a7d0 con 0x7fadd00f54d0 2021-11-20T05:38:12.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:12 smithi187 conmon[36358]: audit 2021-11-20T05:38:11.454315+0000 mon.smithi058 (mon.0) 816 : audit [DBG] from='client.? 172.21.15.58:0/3681151382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:12.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:12 smithi187 conmon[36358]: cluster 2021-11-20T05:38:11.482054+0000 mon.smithi058 (mon.0) 817 : cluster [INF] daemon mds.cephfs.smithi187.fikisa restarted 2021-11-20T05:38:12.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:12 smithi187 conmon[36358]: cluster 2021-11-20T05:38:11.485817+0000 mon.smithi058 (mon.0) 818 2021-11-20T05:38:12.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:12 smithi187 conmon[36358]: : cluster [DBG] osdmap e56: 6 total, 6 up, 6 in 2021-11-20T05:38:12.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:12 smithi058 conmon[33537]: audit 2021-11-20T05:38:11.454315+0000 mon.smithi058 (mon.0) 816 : audit [DBG] from='client.? 172.21.15.58:0/3681151382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:12.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:12 smithi058 conmon[33537]: cluster 2021-11-20T05:38:12.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:12 smithi058 conmon[33537]: 2021-11-20T05:38:11.482054+0000 mon.smithi058 (mon.0) 817 : cluster [INF] daemon mds.cephfs.smithi187.fikisa restarted 2021-11-20T05:38:12.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:12 smithi058 conmon[33537]: cluster 2021-11-20T05:38:11.485817+0000 mon.smithi058 (mon.0) 818 : cluster [DBG] osdmap e56: 6 total, 6 up, 6 in 2021-11-20T05:38:12.800 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.805+0000 7fadd733e700 1 -- 172.21.15.58:0/616458844 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadd0045ad0 con 0x7fadd00f54d0 2021-11-20T05:38:12.801 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.805+0000 7fadc6ffd700 1 -- 172.21.15.58:0/616458844 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2576 (secure 0 0 0) 0x7fadcc04ddf0 con 0x7fadd00f54d0 2021-11-20T05:38:12.801 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:12.802 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:12.302228+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24543,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24543":{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":86,"state":"up:replay","state_seq":2,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:12.803 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.807+0000 7fadd733e700 1 -- 172.21.15.58:0/616458844 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fadbc05c300 msgr2=0x7fadbc05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:12.803 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.807+0000 7fadd733e700 1 --2- 172.21.15.58:0/616458844 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fadbc05c300 0x7fadbc05e7c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fadc0009740 tx=0x7fadc0009be0).stop 2021-11-20T05:38:12.803 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.807+0000 7fadd733e700 1 -- 172.21.15.58:0/616458844 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fadd00f54d0 msgr2=0x7fadd0109aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:12.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.807+0000 7fadd733e700 1 --2- 172.21.15.58:0/616458844 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fadd00f54d0 0x7fadd0109aa0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fadcc004e30 tx=0x7fadcc009b30).stop 2021-11-20T05:38:12.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.807+0000 7fadd733e700 1 -- 172.21.15.58:0/616458844 shutdown_connections 2021-11-20T05:38:12.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.807+0000 7fadd733e700 1 --2- 172.21.15.58:0/616458844 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fadbc05c300 0x7fadbc05e7c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:12.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.807+0000 7fadd733e700 1 --2- 172.21.15.58:0/616458844 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fadd00f54d0 0x7fadd0109aa0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:12.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.807+0000 7fadd733e700 1 --2- 172.21.15.58:0/616458844 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadd00f6770 0x7fadd010bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:12.805 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.807+0000 7fadd733e700 1 -- 172.21.15.58:0/616458844 >> 172.21.15.58:0/616458844 conn(0x7fadd00f0c30 msgr2=0x7fadd00fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:12.805 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.807+0000 7fadd733e700 1 -- 172.21.15.58:0/616458844 shutdown_connections 2021-11-20T05:38:12.805 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:12.807+0000 7fadd733e700 1 -- 172.21.15.58:0/616458844 wait complete. 2021-11-20T05:38:12.806 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 86 2021-11-20T05:38:12.815 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:12.302228+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24543, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24543': {'gid': 24543, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 86, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.187:6825/1821831102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1821831102}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1821831102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:12.815 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24543:up:replay 2021-11-20T05:38:13.574 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:13 smithi187 conmon[36358]: cluster 2021-11-20T05:38:12.296198+0000 mgr.smithi058.zagdlk (mgr.14164) 315 : cluster [DBG] pgmap v291: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 204 B/s wr, 0 op/s 2021-11-20T05:38:13.575 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:13 smithi187 conmon[36358]: 2021-11-20T05:38:13.575 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:13 smithi187 conmon[36358]: cluster 2021-11-20T05:38:12.302078+0000 mon.smithi058 (mon.0) 819 : cluster [DBG] mds.? [v2:172.21.15.187:6826/1745887865,v1:172.21.15.187:6827/1745887865] up:boot 2021-11-20T05:38:13.575 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:13 smithi187 conmon[36358]: cluster 2021-11-20T05:38:12.302171+0000 mon.smithi058 (mon 2021-11-20T05:38:13.575 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:13 smithi187 conmon[36358]: .0) 820 : cluster [INF] Standby daemon mds.cephfs.smithi187.wxduhu assigned to filesystem cephfs as rank 0 2021-11-20T05:38:13.576 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:13 smithi187 conmon[36358]: cluster 2021-11-20T05: 2021-11-20T05:38:13.576 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:13 smithi187 conmon[36358]: 38:12.302507+0000 mon.smithi058 (mon.0) 821 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby, 1 failed 2021-11-20T05:38:13.576 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:13 smithi187 conmon[36358]: audit 2021-11 2021-11-20T05:38:13.576 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:13 smithi187 conmon[36358]: -20T05:38:12.302635+0000 mon.smithi058 ( 2021-11-20T05:38:13.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:13 smithi187 conmon[36358]: mon.0) 822 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.fikisa"}]: dispatch 2021-11-20T05:38:13.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:13 smithi187 conmon[36358]: cluster 2021-11 2021-11-20T05:38:13.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:13 smithi187 conmon[36358]: -20T05:38:12.306554+0000 mon.smithi058 ( 2021-11-20T05:38:13.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:13 smithi187 conmon[36358]: mon.0) 823 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:13.577 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:13 smithi187 conmon[36358]: audit 2021- 2021-11-20T05:38:13.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:13 smithi187 conmon[36358]: 11-20T05:38:12.806163+0000 mon.smithi058 2021-11-20T05:38:13.578 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:13 smithi187 conmon[36358]: (mon.0) 824 : audit [DBG] from='client.? 172.21.15.58:0/616458844' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:13.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: cluster 2021-11-20T05:38:12.296198+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:38:13.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: 315 : cluster [DBG] pgmap v291: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 204 B/s wr, 0 op/s 2021-11-20T05:38:13.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: cluster 2021-11-20T05:38:12.302078+0000 mon.smithi058 (mon.0) 819 : cluster [DBG] mds.? [v2:172.21.15.187:6826/1745887865,v1:172.21.15.187:6827/1745887865] up:boot 2021-11-20T05:38:13.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: cluster 2021-11-20T05:38:12.302171+0000 mon.smithi058 (mon.0) 820 : 2021-11-20T05:38:13.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: cluster [INF] Standby daemon mds.cephfs.smithi187.wxduhu assigned to filesystem cephfs as rank 0 2021-11-20T05:38:13.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: cluster 2021-11-20T05:38:13.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: 2021-11-20T05: 2021-11-20T05:38:13.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: 38:12.302507+0000 mon.smithi058 ( 2021-11-20T05:38:13.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: mon.0) 821 : cluster [DBG] 2021-11-20T05:38:13.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby, 1 failed 2021-11-20T05:38:13.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: audit 2021-11-20T05:38:13.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: 2021-11-20T05:38:12. 2021-11-20T05:38:13.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: 302635+0000 mon.smithi058 (mon.0) 822 : audit 2021-11-20T05:38:13.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.fikisa"}]: dispatch 2021-11-20T05:38:13.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: cluster 2021-11-20T05:38: 2021-11-20T05:38:13.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: 12.306554+0000 mon.smithi058 (mon.0) 2021-11-20T05:38:13.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: 823 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:13.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: audit 2021-11-20T 2021-11-20T05:38:13.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: 05:38:12.806163+0000 mon.smithi058 (mon 2021-11-20T05:38:13.721 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:13 smithi058 conmon[33537]: .0) 824 : audit [DBG] from='client.? 172.21.15.58:0/616458844' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:13.816 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:13.950 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.952+0000 7fa33610a700 1 -- 172.21.15.58:0/1407858856 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa3300f5500 msgr2=0x7fa3300f5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:13.950 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.952+0000 7fa33610a700 1 --2- 172.21.15.58:0/1407858856 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa3300f5500 0x7fa3300f5920 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fa320004660 tx=0x7fa320009b30).stop 2021-11-20T05:38:13.951 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.952+0000 7fa33610a700 1 -- 172.21.15.58:0/1407858856 shutdown_connections 2021-11-20T05:38:13.951 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.952+0000 7fa33610a700 1 --2- 172.21.15.58:0/1407858856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3300f67a0 0x7fa3300f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:13.951 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.952+0000 7fa33610a700 1 --2- 172.21.15.58:0/1407858856 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa3300f5500 0x7fa3300f5920 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:13.951 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.952+0000 7fa33610a700 1 -- 172.21.15.58:0/1407858856 >> 172.21.15.58:0/1407858856 conn(0x7fa3300f0c40 msgr2=0x7fa3300f3080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:13.952 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.952+0000 7fa33610a700 1 -- 172.21.15.58:0/1407858856 shutdown_connections 2021-11-20T05:38:13.952 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.953+0000 7fa33610a700 1 -- 172.21.15.58:0/1407858856 wait complete. 2021-11-20T05:38:13.952 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.953+0000 7fa33610a700 1 Processor -- start 2021-11-20T05:38:13.952 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.953+0000 7fa33610a700 1 -- start start 2021-11-20T05:38:13.952 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.953+0000 7fa33610a700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa3300f5500 0x7fa330109a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:13.953 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.953+0000 7fa33610a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3300f67a0 0x7fa33010bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:13.953 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.953+0000 7fa33610a700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa330060560 con 0x7fa3300f5500 2021-11-20T05:38:13.953 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.953+0000 7fa33610a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3300606a0 con 0x7fa3300f67a0 2021-11-20T05:38:13.953 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.953+0000 7fa335108700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa3300f5500 0x7fa330109a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:13.953 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.953+0000 7fa335108700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa3300f5500 0x7fa330109a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42058/0 (socket says 172.21.15.58:42058) 2021-11-20T05:38:13.954 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.953+0000 7fa335108700 1 -- 172.21.15.58:0/4255883630 learned_addr learned my addr 172.21.15.58:0/4255883630 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:13.954 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.953+0000 7fa334907700 1 --2- 172.21.15.58:0/4255883630 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3300f67a0 0x7fa33010bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:13.954 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.954+0000 7fa335108700 1 -- 172.21.15.58:0/4255883630 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3300f67a0 msgr2=0x7fa33010bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:13.954 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.954+0000 7fa335108700 1 --2- 172.21.15.58:0/4255883630 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3300f67a0 0x7fa33010bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:13.955 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.954+0000 7fa335108700 1 -- 172.21.15.58:0/4255883630 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa320005040 con 0x7fa3300f5500 2021-11-20T05:38:13.955 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.954+0000 7fa335108700 1 --2- 172.21.15.58:0/4255883630 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa3300f5500 0x7fa330109a60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fa320004e30 tx=0x7fa320009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:13.955 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.954+0000 7fa325ffb700 1 -- 172.21.15.58:0/4255883630 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa320018070 con 0x7fa3300f5500 2021-11-20T05:38:13.955 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.954+0000 7fa33610a700 1 -- 172.21.15.58:0/4255883630 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa33010c4b0 con 0x7fa3300f5500 2021-11-20T05:38:13.956 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.954+0000 7fa325ffb700 1 -- 172.21.15.58:0/4255883630 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa320013410 con 0x7fa3300f5500 2021-11-20T05:38:13.956 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.954+0000 7fa325ffb700 1 -- 172.21.15.58:0/4255883630 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa32001de20 con 0x7fa3300f5500 2021-11-20T05:38:13.956 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.954+0000 7fa33610a700 1 -- 172.21.15.58:0/4255883630 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33010ca10 con 0x7fa3300f5500 2021-11-20T05:38:13.958 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.955+0000 7fa33610a700 1 -- 172.21.15.58:0/4255883630 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa330045ad0 con 0x7fa3300f5500 2021-11-20T05:38:13.958 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.956+0000 7fa325ffb700 1 -- 172.21.15.58:0/4255883630 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa32001d420 con 0x7fa3300f5500 2021-11-20T05:38:13.958 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.956+0000 7fa325ffb700 1 --2- 172.21.15.58:0/4255883630 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa31c05c300 0x7fa31c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:13.958 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.956+0000 7fa325ffb700 1 -- 172.21.15.58:0/4255883630 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa320077ff0 con 0x7fa3300f5500 2021-11-20T05:38:13.959 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.956+0000 7fa334907700 1 --2- 172.21.15.58:0/4255883630 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa31c05c300 0x7fa31c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:13.959 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.957+0000 7fa334907700 1 --2- 172.21.15.58:0/4255883630 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa31c05c300 0x7fa31c05e7c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fa32c004c40 tx=0x7fa32c00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:13.959 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:13.960+0000 7fa325ffb700 1 -- 172.21.15.58:0/4255883630 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa32004a800 con 0x7fa3300f5500 2021-11-20T05:38:14.149 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:14.154+0000 7fa33610a700 1 -- 172.21.15.58:0/4255883630 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3300fdd60 con 0x7fa3300f5500 2021-11-20T05:38:14.150 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:14.154+0000 7fa325ffb700 1 -- 172.21.15.58:0/4255883630 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2577 (secure 0 0 0) 0x7fa32004de20 con 0x7fa3300f5500 2021-11-20T05:38:14.151 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:14.151 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:13.304827+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24543,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24543":{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":86,"state":"up:resolve","state_seq":6,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:14.152 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:14.156+0000 7fa33610a700 1 -- 172.21.15.58:0/4255883630 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa31c05c300 msgr2=0x7fa31c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:14.153 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:14.156+0000 7fa33610a700 1 --2- 172.21.15.58:0/4255883630 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa31c05c300 0x7fa31c05e7c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fa32c004c40 tx=0x7fa32c00b040).stop 2021-11-20T05:38:14.153 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:14.156+0000 7fa33610a700 1 -- 172.21.15.58:0/4255883630 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa3300f5500 msgr2=0x7fa330109a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:14.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:14.156+0000 7fa33610a700 1 --2- 172.21.15.58:0/4255883630 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa3300f5500 0x7fa330109a60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fa320004e30 tx=0x7fa320009b30).stop 2021-11-20T05:38:14.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:14.156+0000 7fa33610a700 1 -- 172.21.15.58:0/4255883630 shutdown_connections 2021-11-20T05:38:14.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:14.156+0000 7fa33610a700 1 --2- 172.21.15.58:0/4255883630 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa31c05c300 0x7fa31c05e7c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:14.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:14.156+0000 7fa33610a700 1 --2- 172.21.15.58:0/4255883630 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa3300f5500 0x7fa330109a60 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:14.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:14.156+0000 7fa33610a700 1 --2- 172.21.15.58:0/4255883630 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3300f67a0 0x7fa33010bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:14.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:14.156+0000 7fa33610a700 1 -- 172.21.15.58:0/4255883630 >> 172.21.15.58:0/4255883630 conn(0x7fa3300f0c40 msgr2=0x7fa3300fa1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:14.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:14.156+0000 7fa33610a700 1 -- 172.21.15.58:0/4255883630 shutdown_connections 2021-11-20T05:38:14.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:14.156+0000 7fa33610a700 1 -- 172.21.15.58:0/4255883630 wait complete. 2021-11-20T05:38:14.157 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 87 2021-11-20T05:38:14.164 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:13.304827+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24543, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24543': {'gid': 24543, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 86, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.187:6825/1821831102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1821831102}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1821831102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:14.165 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24543:up:resolve 2021-11-20T05:38:14.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:14 smithi058 conmon[33537]: cluster 2021-11-20T05:38: 2021-11-20T05:38:14.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:14 smithi058 conmon[33537]: 13.308427+0000 mon.smithi058 (mon.0) 825 : cluster [DBG] mds.? [v2:172.21.15.187:6824/1821831102,v1:172.21.15.187:6825/1821831102] up:resolve 2021-11-20T05:38:14.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:14 smithi058 conmon[33537]: cluster 2021-11-20T05:38:13. 2021-11-20T05:38:14.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:14 smithi058 conmon[33537]: 308504+0000 mon.smithi058 (mon.0) 826 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:14.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:14 smithi058 conmon[33537]: audit 2021-11-20T05:38 2021-11-20T05:38:14.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:14 smithi058 conmon[33537]: :14.155438+0000 mon.smithi058 (mon.0) 827 2021-11-20T05:38:14.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:14 smithi058 conmon[33537]: : audit [DBG] from='client.? 172.21.15.58:0/4255883630' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:14.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:14 smithi187 conmon[36358]: cluster 2021-11-20T05:38:13.308427+0000 mon.smithi058 (mon.0) 825 : cluster [DBG] mds.? [v2:172.21.15.187:6824/1821831102,v1:172.21.15.187:6825/1821831102] up:resolve 2021-11-20T05:38:14.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:14 smithi187 conmon[36358]: cluster 2021-11-20T05:38:13.308504+0000 mon.smithi058 (mon.0) 826 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:14.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:14 smithi187 conmon[36358]: audit 2021-11-20T05:38:14.155438+0000 mon.smithi058 (mon.0) 827 : audit [DBG] from='client.? 172.21.15.58:0/4255883630' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:15.166 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:15.318 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.320+0000 7f637c7a1700 1 -- 172.21.15.58:0/4017451790 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63740f6840 msgr2=0x7f63740f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:15.319 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.320+0000 7f637c7a1700 1 --2- 172.21.15.58:0/4017451790 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63740f6840 0x7f63740f6ca0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f6364004660 tx=0x7f6364009b30).stop 2021-11-20T05:38:15.319 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.320+0000 7f637c7a1700 1 -- 172.21.15.58:0/4017451790 shutdown_connections 2021-11-20T05:38:15.319 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.320+0000 7f637c7a1700 1 --2- 172.21.15.58:0/4017451790 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63740f6840 0x7f63740f6ca0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:15.319 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.321+0000 7f637c7a1700 1 --2- 172.21.15.58:0/4017451790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63740f55a0 0x7f63740f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:15.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.321+0000 7f637c7a1700 1 -- 172.21.15.58:0/4017451790 >> 172.21.15.58:0/4017451790 conn(0x7f63740f0ce0 msgr2=0x7f63740f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:15.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.321+0000 7f637c7a1700 1 -- 172.21.15.58:0/4017451790 shutdown_connections 2021-11-20T05:38:15.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.321+0000 7f637c7a1700 1 -- 172.21.15.58:0/4017451790 wait complete. 2021-11-20T05:38:15.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.321+0000 7f637c7a1700 1 Processor -- start 2021-11-20T05:38:15.320 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.321+0000 7f637c7a1700 1 -- start start 2021-11-20T05:38:15.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.321+0000 7f637c7a1700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63740f55a0 0x7f637405ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:15.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.321+0000 7f637c7a1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63740f6840 0x7f637410bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:15.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.321+0000 7f637c7a1700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637410c130 con 0x7f63740f55a0 2021-11-20T05:38:15.321 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.321+0000 7f637c7a1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637410c270 con 0x7f63740f6840 2021-11-20T05:38:15.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.321+0000 7f637a53d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63740f55a0 0x7f637405ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:15.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.321+0000 7f637a53d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63740f55a0 0x7f637405ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42072/0 (socket says 172.21.15.58:42072) 2021-11-20T05:38:15.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.321+0000 7f637a53d700 1 -- 172.21.15.58:0/2450537387 learned_addr learned my addr 172.21.15.58:0/2450537387 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:15.322 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.321+0000 7f6379d3c700 1 --2- 172.21.15.58:0/2450537387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63740f6840 0x7f637410bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:15.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.322+0000 7f637a53d700 1 -- 172.21.15.58:0/2450537387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63740f6840 msgr2=0x7f637410bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:15.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.322+0000 7f637a53d700 1 --2- 172.21.15.58:0/2450537387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63740f6840 0x7f637410bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:15.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.322+0000 7f637a53d700 1 -- 172.21.15.58:0/2450537387 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6364005040 con 0x7f63740f55a0 2021-11-20T05:38:15.323 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.322+0000 7f637a53d700 1 --2- 172.21.15.58:0/2450537387 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63740f55a0 0x7f637405ff00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f637000e7c0 tx=0x7f637000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:15.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.322+0000 7f636affd700 1 -- 172.21.15.58:0/2450537387 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63700092e0 con 0x7f63740f55a0 2021-11-20T05:38:15.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.322+0000 7f637c7a1700 1 -- 172.21.15.58:0/2450537387 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f637410c4d0 con 0x7f63740f55a0 2021-11-20T05:38:15.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.322+0000 7f636affd700 1 -- 172.21.15.58:0/2450537387 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f637000f040 con 0x7f63740f55a0 2021-11-20T05:38:15.324 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.322+0000 7f636affd700 1 -- 172.21.15.58:0/2450537387 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63700147c0 con 0x7f63740f55a0 2021-11-20T05:38:15.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.322+0000 7f637c7a1700 1 -- 172.21.15.58:0/2450537387 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f637410ca60 con 0x7f63740f55a0 2021-11-20T05:38:15.325 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.323+0000 7f637c7a1700 1 -- 172.21.15.58:0/2450537387 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6374027a00 con 0x7f63740f55a0 2021-11-20T05:38:15.326 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.323+0000 7f636affd700 1 -- 172.21.15.58:0/2450537387 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f637001b030 con 0x7f63740f55a0 2021-11-20T05:38:15.326 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.324+0000 7f636affd700 1 --2- 172.21.15.58:0/2450537387 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f636005c2a0 0x7f636005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:15.326 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.324+0000 7f636affd700 1 -- 172.21.15.58:0/2450537387 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f6370076300 con 0x7f63740f55a0 2021-11-20T05:38:15.327 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.324+0000 7f6379d3c700 1 --2- 172.21.15.58:0/2450537387 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f636005c2a0 0x7f636005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:15.327 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.324+0000 7f6379d3c700 1 --2- 172.21.15.58:0/2450537387 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f636005c2a0 0x7f636005e760 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f6364005010 tx=0x7f6364016040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:15.327 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.328+0000 7f636affd700 1 -- 172.21.15.58:0/2450537387 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6370048b10 con 0x7f63740f55a0 2021-11-20T05:38:15.465 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:15 smithi058 conmon[33537]: cluster 2021-11-20T05:38:14.296645+0000 mgr.smithi058.zagdlk 2021-11-20T05:38:15.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:15 smithi058 conmon[33537]: (mgr.14164) 316 : cluster [DBG] pgmap v292: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:38:15.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:15 smithi058 conmon[33537]: cluster 2021-11-20T05:38:14.308972+0000 mon.smithi058 ( 2021-11-20T05:38:15.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:15 smithi058 conmon[33537]: mon.0) 828 : cluster [DBG] mds.? [v2:172.21.15.187:6824/1821831102,v1:172.21.15.187:6825/1821831102] up:reconnect 2021-11-20T05:38:15.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:15 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:38:15.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:15 smithi058 conmon[33537]: -11-20T05:38: 2021-11-20T05:38:15.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:15 smithi058 conmon[33537]: 14.309053+0000 mon.smithi058 (mon. 2021-11-20T05:38:15.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:15 smithi058 conmon[33537]: 0) 829 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:15.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:15 smithi058 conmon[33537]: cluster 2021- 2021-11-20T05:38:15.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:15 smithi058 conmon[33537]: 11-20T05:38:14.329598+0000 mds.cephfs.smithi187.wxduhu 2021-11-20T05:38:15.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:15 smithi058 conmon[33537]: (mds.0) 1 : cluster [DBG] 2021-11-20T05:38:15.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:15 smithi058 conmon[33537]: reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0219996 2021-11-20T05:38:15.469 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:15 smithi187 conmon[36358]: cluster 2021-11-20T05:38:14.296645+0000 mgr.smithi058.zagdlk (mgr 2021-11-20T05:38:15.469 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:15 smithi187 conmon[36358]: .14164) 316 : cluster [DBG] pgmap v292: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:38:15.469 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:15 smithi187 conmon[36358]: cluster 2021-11-20T05:38:14.308972+0000 mon.smithi058 (mon.0) 828 : cluster [DBG] mds.? [v2:172.21.15.187:6824/1821831102,v1:172.21.15.187:6825/1821831102] up:reconnect 2021-11-20T05:38:15.469 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:15 smithi187 conmon[36358]: 2021-11-20T05:38:15.470 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:15 smithi187 conmon[36358]: cluster 2021-11- 2021-11-20T05:38:15.470 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:15 smithi187 conmon[36358]: 20T05:38:14.309053+0000 2021-11-20T05:38:15.470 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:15 smithi187 conmon[36358]: mon.smithi058 (mon.0) 829 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:15.470 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:15 smithi187 conmon[36358]: 2021-11-20T05:38:15.471 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:15 smithi187 conmon[36358]: cluster 2021-11-20T05:38:15.471 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:15 smithi187 conmon[36358]: 2021-11-20T05:38:14 2021-11-20T05:38:15.471 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:15 smithi187 conmon[36358]: .329598+0000 mds.cephfs.smithi187.wxduhu (mds.0) 1 : cluster 2021-11-20T05:38:15.471 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:15 smithi187 conmon[36358]: [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0219996 2021-11-20T05:38:15.520 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.525+0000 7f637c7a1700 1 -- 172.21.15.58:0/2450537387 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63740fdd30 con 0x7f63740f55a0 2021-11-20T05:38:15.521 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.525+0000 7f636affd700 1 -- 172.21.15.58:0/2450537387 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+2576 (secure 0 0 0) 0x7f6370019020 con 0x7f63740f55a0 2021-11-20T05:38:15.522 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:15.522 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":85}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:15.310544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24543,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24543":{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:15.523 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.527+0000 7f637c7a1700 1 -- 172.21.15.58:0/2450537387 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f636005c2a0 msgr2=0x7f636005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:15.524 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.527+0000 7f637c7a1700 1 --2- 172.21.15.58:0/2450537387 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f636005c2a0 0x7f636005e760 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f6364005010 tx=0x7f6364016040).stop 2021-11-20T05:38:15.524 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.527+0000 7f637c7a1700 1 -- 172.21.15.58:0/2450537387 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63740f55a0 msgr2=0x7f637405ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:15.524 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.527+0000 7f637c7a1700 1 --2- 172.21.15.58:0/2450537387 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63740f55a0 0x7f637405ff00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f637000e7c0 tx=0x7f637000b040).stop 2021-11-20T05:38:15.524 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.527+0000 7f637c7a1700 1 -- 172.21.15.58:0/2450537387 shutdown_connections 2021-11-20T05:38:15.525 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.527+0000 7f637c7a1700 1 --2- 172.21.15.58:0/2450537387 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f636005c2a0 0x7f636005e760 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:15.525 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.527+0000 7f637c7a1700 1 --2- 172.21.15.58:0/2450537387 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f63740f55a0 0x7f637405ff00 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:15.525 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.527+0000 7f637c7a1700 1 --2- 172.21.15.58:0/2450537387 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63740f6840 0x7f637410bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:15.525 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.527+0000 7f637c7a1700 1 -- 172.21.15.58:0/2450537387 >> 172.21.15.58:0/2450537387 conn(0x7f63740f0ce0 msgr2=0x7f63740fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:15.526 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.527+0000 7f637c7a1700 1 -- 172.21.15.58:0/2450537387 shutdown_connections 2021-11-20T05:38:15.526 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:15.527+0000 7f637c7a1700 1 -- 172.21.15.58:0/2450537387 wait complete. 2021-11-20T05:38:15.526 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 89 2021-11-20T05:38:15.535 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:15.310544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24543, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24543': {'gid': 24543, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 86, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6825/1821831102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1821831102}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1821831102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:15.535 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24543:up:rejoin 2021-11-20T05:38:16.528 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:16 smithi058 conmon[33537]: cluster 2021-11-20T05:38:15.318511+0000 mon.smithi058 (mon.0) 830 : 2021-11-20T05:38:16.528 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:16 smithi058 conmon[33537]: cluster [DBG] mds.? [v2:172.21.15.187:6824/1821831102,v1:172.21.15.187:6825/1821831102] up:rejoin 2021-11-20T05:38:16.529 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:16 smithi058 conmon[33537]: cluster 2021-11-20T05:38:15.318586+0000 mon.smithi058 (mon.0) 831 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:16.529 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:16 smithi058 conmon[33537]: audit 2021-11-20T05:38:16.529 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:16 smithi058 conmon[33537]: 2021-11-20T05:38:15. 2021-11-20T05:38:16.529 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:16 smithi058 conmon[33537]: 526382+0000 mon.smithi058 (mon.0) 2021-11-20T05:38:16.530 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:16 smithi058 conmon[33537]: 832 : audit [DBG] from='client.? 172.21.15.58:0/2450537387' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:16.536 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:16.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.672+0000 7f820079d700 1 -- 172.21.15.58:0/999172897 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f81f80f5320 msgr2=0x7f81f80f7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:16.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.672+0000 7f820079d700 1 --2- 172.21.15.58:0/999172897 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f81f80f5320 0x7f81f80f7aa0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f81ec004660 tx=0x7f81ec009b30).stop 2021-11-20T05:38:16.672 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.672+0000 7f820079d700 1 -- 172.21.15.58:0/999172897 shutdown_connections 2021-11-20T05:38:16.673 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.672+0000 7f820079d700 1 --2- 172.21.15.58:0/999172897 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f81f80f5320 0x7f81f80f7aa0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:16.673 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.672+0000 7f820079d700 1 --2- 172.21.15.58:0/999172897 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81f80f4980 0x7f81f80f4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:16.673 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.672+0000 7f820079d700 1 -- 172.21.15.58:0/999172897 >> 172.21.15.58:0/999172897 conn(0x7f81f80f04a0 msgr2=0x7f81f80f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:16.674 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.672+0000 7f820079d700 1 -- 172.21.15.58:0/999172897 shutdown_connections 2021-11-20T05:38:16.674 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.672+0000 7f820079d700 1 -- 172.21.15.58:0/999172897 wait complete. 2021-11-20T05:38:16.674 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.673+0000 7f820079d700 1 Processor -- start 2021-11-20T05:38:16.674 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.673+0000 7f820079d700 1 -- start start 2021-11-20T05:38:16.675 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.673+0000 7f820079d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f81f80f4980 0x7f81f8106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:16.675 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.673+0000 7f820079d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81f80f5320 0x7f81f81073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:16.675 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.673+0000 7f820079d700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f8107970 con 0x7f81f80f4980 2021-11-20T05:38:16.675 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.673+0000 7f820079d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f8107ab0 con 0x7f81f80f5320 2021-11-20T05:38:16.676 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.673+0000 7f81fe539700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f81f80f4980 0x7f81f8106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:16.676 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.673+0000 7f81fe539700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f81f80f4980 0x7f81f8106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42090/0 (socket says 172.21.15.58:42090) 2021-11-20T05:38:16.676 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.673+0000 7f81fe539700 1 -- 172.21.15.58:0/2646983662 learned_addr learned my addr 172.21.15.58:0/2646983662 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:16.676 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.673+0000 7f81fdd38700 1 --2- 172.21.15.58:0/2646983662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81f80f5320 0x7f81f81073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:16.676 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.673+0000 7f81fe539700 1 -- 172.21.15.58:0/2646983662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81f80f5320 msgr2=0x7f81f81073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:16.677 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.673+0000 7f81fe539700 1 --2- 172.21.15.58:0/2646983662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81f80f5320 0x7f81f81073c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:16.677 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.673+0000 7f81fe539700 1 -- 172.21.15.58:0/2646983662 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ec005040 con 0x7f81f80f4980 2021-11-20T05:38:16.677 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.673+0000 7f81fe539700 1 --2- 172.21.15.58:0/2646983662 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f81f80f4980 0x7f81f8106ed0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f81f0002700 tx=0x7f81f0009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:16.677 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.674+0000 7f81f6ffd700 1 -- 172.21.15.58:0/2646983662 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81f0009320 con 0x7f81f80f4980 2021-11-20T05:38:16.678 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.674+0000 7f81f6ffd700 1 -- 172.21.15.58:0/2646983662 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81f000f040 con 0x7f81f80f4980 2021-11-20T05:38:16.678 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.674+0000 7f820079d700 1 -- 172.21.15.58:0/2646983662 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f810c550 con 0x7f81f80f4980 2021-11-20T05:38:16.678 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.674+0000 7f81f6ffd700 1 -- 172.21.15.58:0/2646983662 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81f0014900 con 0x7f81f80f4980 2021-11-20T05:38:16.678 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.674+0000 7f820079d700 1 -- 172.21.15.58:0/2646983662 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f810cb10 con 0x7f81f80f4980 2021-11-20T05:38:16.678 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.675+0000 7f820079d700 1 -- 172.21.15.58:0/2646983662 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f8027a00 con 0x7f81f80f4980 2021-11-20T05:38:16.679 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.675+0000 7f81f6ffd700 1 -- 172.21.15.58:0/2646983662 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f81f0009480 con 0x7f81f80f4980 2021-11-20T05:38:16.679 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.675+0000 7f81f6ffd700 1 --2- 172.21.15.58:0/2646983662 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f81e005c2a0 0x7f81e005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:16.679 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.675+0000 7f81f6ffd700 1 -- 172.21.15.58:0/2646983662 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f81f0076790 con 0x7f81f80f4980 2021-11-20T05:38:16.679 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.675+0000 7f81fdd38700 1 --2- 172.21.15.58:0/2646983662 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f81e005c2a0 0x7f81e005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:16.680 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.676+0000 7f81fdd38700 1 --2- 172.21.15.58:0/2646983662 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f81e005c2a0 0x7f81e005e760 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f81ec004e30 tx=0x7f81ec009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:16.680 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.679+0000 7f81f6ffd700 1 -- 172.21.15.58:0/2646983662 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f81f0048ee0 con 0x7f81f80f4980 2021-11-20T05:38:16.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:16 smithi187 conmon[36358]: cluster 2021-11-20T05:38:15.318511+0000 mon.smithi058 (mon.0) 830 : cluster [DBG] mds.? [v2:172.21.15.187:6824/1821831102,v1:172.21.15.187:6825/1821831102] up:rejoin 2021-11-20T05:38:16.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:16 smithi187 conmon[36358]: cluster 2021-11-20T05:38:15.318586+0000 mon.smithi058 (mon.0) 831 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:16.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:16 smithi187 conmon[36358]: audit 2021-11-20T05: 2021-11-20T05:38:16.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:16 smithi187 conmon[36358]: 38:15.526382+0000 mon.smithi058 (mon.0) 832 : audit [DBG] from='client.? 172.21.15.58:0/2450537387' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:16.870 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.874+0000 7f820079d700 1 -- 172.21.15.58:0/2646983662 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81f80fee70 con 0x7f81f80f4980 2021-11-20T05:38:16.870 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.875+0000 7f81f6ffd700 1 -- 172.21.15.58:0/2646983662 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f81f004c500 con 0x7f81f80f4980 2021-11-20T05:38:16.871 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:16.871 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:15.310544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24543,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24543":{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:16.872 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.876+0000 7f820079d700 1 -- 172.21.15.58:0/2646983662 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f81e005c2a0 msgr2=0x7f81e005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:16.872 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.876+0000 7f820079d700 1 --2- 172.21.15.58:0/2646983662 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f81e005c2a0 0x7f81e005e760 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f81ec004e30 tx=0x7f81ec009b30).stop 2021-11-20T05:38:16.873 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.876+0000 7f820079d700 1 -- 172.21.15.58:0/2646983662 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f81f80f4980 msgr2=0x7f81f8106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:16.873 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.876+0000 7f820079d700 1 --2- 172.21.15.58:0/2646983662 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f81f80f4980 0x7f81f8106ed0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f81f0002700 tx=0x7f81f0009d80).stop 2021-11-20T05:38:16.873 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.876+0000 7f820079d700 1 -- 172.21.15.58:0/2646983662 shutdown_connections 2021-11-20T05:38:16.873 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.876+0000 7f820079d700 1 --2- 172.21.15.58:0/2646983662 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f81e005c2a0 0x7f81e005e760 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:16.874 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.876+0000 7f820079d700 1 --2- 172.21.15.58:0/2646983662 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f81f80f4980 0x7f81f8106ed0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:16.874 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.876+0000 7f820079d700 1 --2- 172.21.15.58:0/2646983662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81f80f5320 0x7f81f81073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:16.874 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.876+0000 7f820079d700 1 -- 172.21.15.58:0/2646983662 >> 172.21.15.58:0/2646983662 conn(0x7f81f80f04a0 msgr2=0x7f81f80fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:16.874 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.876+0000 7f820079d700 1 -- 172.21.15.58:0/2646983662 shutdown_connections 2021-11-20T05:38:16.875 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:16.876+0000 7f820079d700 1 -- 172.21.15.58:0/2646983662 wait complete. 2021-11-20T05:38:16.875 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 90 2021-11-20T05:38:16.884 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:15.310544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24543, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24543': {'gid': 24543, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 86, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6825/1821831102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1821831102}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1821831102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:16.884 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24543:up:rejoin 2021-11-20T05:38:17.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:17 smithi187 conmon[36358]: cluster 2021-11-20T05:38:16.297111+0000 mgr.smithi058.zagdlk (mgr.14164) 317 : cluster [DBG] pgmap v293: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:38:17.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:17 smithi187 conmon[36358]: cluster 2021-11-20T05:38:16.323291+0000 mon.smithi058 (mon.0) 833 : cluster [DBG] mds.? [v2:172.21.15.187:6826/1745887865,v1:172.21.15.187:6827/1745887865] up:standby 2021-11-20T05:38:17.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:17 smithi187 conmon[36358]: cluster 2021-11-20T05:38:16.323416+0000 mon.smithi058 (mon.0) 834 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:17.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:17 smithi187 conmon[36358]: audit 2021-11-20T05:38:16 2021-11-20T05:38:17.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:17 smithi187 conmon[36358]: .875582+0000 mon.smithi058 (mon.0) 835 : audit [DBG] 2021-11-20T05:38:17.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:17 smithi187 conmon[36358]: from='client.? 172.21.15.58:0/2646983662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:17.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:17 smithi058 conmon[33537]: cluster 2021-11-20T05:38:16.297111+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:38:17.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:17 smithi058 conmon[33537]: 14164) 317 : cluster [DBG] pgmap v293: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:38:17.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:17 smithi058 conmon[33537]: cluster 2021-11-20T05:38:16.323291+0000 mon.smithi058 (mon.0) 833 : cluster 2021-11-20T05:38:17.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:17 smithi058 conmon[33537]: [DBG] mds.? [v2:172.21.15.187:6826/1745887865,v1:172.21.15.187:6827/1745887865] up:standby 2021-11-20T05:38:17.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:17 smithi058 conmon[33537]: cluster 2021-11-20T05:38:17.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:17 smithi058 conmon[33537]: 2021-11-20T05:38:16.323416+0000 2021-11-20T05:38:17.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:17 smithi058 conmon[33537]: mon.smithi058 (mon.0) 834 : cluster 2021-11-20T05:38:17.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:17 smithi058 conmon[33537]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:17.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:17 smithi058 conmon[33537]: 2021-11-20T05:38:17.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:17 smithi058 conmon[33537]: audit 2021-11 2021-11-20T05:38:17.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:17 smithi058 conmon[33537]: -20T05:38:16.875582+0000 mon.smithi058 ( 2021-11-20T05:38:17.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:17 smithi058 conmon[33537]: mon.0) 835 : audit [DBG] 2021-11-20T05:38:17.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:17 smithi058 conmon[33537]: from='client.? 172.21.15.58:0/2646983662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:17.885 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:18.021 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.022+0000 7f2954342700 1 -- 172.21.15.58:0/952846184 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f294c0f55a0 msgr2=0x7f294c0f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:18.022 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.022+0000 7f2954342700 1 --2- 172.21.15.58:0/952846184 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f294c0f55a0 0x7f294c0f59c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f29440099e0 tx=0x7f2944004d10).stop 2021-11-20T05:38:18.022 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.022+0000 7f2954342700 1 -- 172.21.15.58:0/952846184 shutdown_connections 2021-11-20T05:38:18.022 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.022+0000 7f2954342700 1 --2- 172.21.15.58:0/952846184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f294c0f6840 0x7f294c0f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:38:18.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.022+0000 7f2954342700 1 --2- 172.21.15.58:0/952846184 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f294c0f55a0 0x7f294c0f59c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:18.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.022+0000 7f2954342700 1 -- 172.21.15.58:0/952846184 >> 172.21.15.58:0/952846184 conn(0x7f294c0f0ce0 msgr2=0x7f294c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:18.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.022+0000 7f2954342700 1 -- 172.21.15.58:0/952846184 shutdown_connections 2021-11-20T05:38:18.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.022+0000 7f2954342700 1 -- 172.21.15.58:0/952846184 wait complete. 2021-11-20T05:38:18.023 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.022+0000 7f2954342700 1 Processor -- start 2021-11-20T05:38:18.024 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.022+0000 7f2954342700 1 -- start start 2021-11-20T05:38:18.024 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.023+0000 7f2954342700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f294c0f55a0 0x7f294c05ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:18.024 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.023+0000 7f2954342700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f294c0f6840 0x7f294c10be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:18.024 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.023+0000 7f2954342700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c10c380 con 0x7f294c0f55a0 2021-11-20T05:38:18.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.023+0000 7f2954342700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c10c4c0 con 0x7f294c0f6840 2021-11-20T05:38:18.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.023+0000 7f29520de700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f294c0f55a0 0x7f294c05ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:18.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.023+0000 7f29520de700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f294c0f55a0 0x7f294c05ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42104/0 (socket says 172.21.15.58:42104) 2021-11-20T05:38:18.025 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.023+0000 7f29520de700 1 -- 172.21.15.58:0/231976759 learned_addr learned my addr 172.21.15.58:0/231976759 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:18.026 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.023+0000 7f29520de700 1 -- 172.21.15.58:0/231976759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f294c0f6840 msgr2=0x7f294c10be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:18.026 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.023+0000 7f29520de700 1 --2- 172.21.15.58:0/231976759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f294c0f6840 0x7f294c10be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:38:18.026 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.023+0000 7f29520de700 1 -- 172.21.15.58:0/231976759 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2944009640 con 0x7f294c0f55a0 2021-11-20T05:38:18.026 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.023+0000 7f29520de700 1 --2- 172.21.15.58:0/231976759 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f294c0f55a0 0x7f294c05ff20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f2944005010 tx=0x7f2944004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:18.026 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.023+0000 7f294affd700 1 -- 172.21.15.58:0/231976759 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f294400e030 con 0x7f294c0f55a0 2021-11-20T05:38:18.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.023+0000 7f294affd700 1 -- 172.21.15.58:0/231976759 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2944007dd0 con 0x7f294c0f55a0 2021-11-20T05:38:18.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.023+0000 7f2954342700 1 -- 172.21.15.58:0/231976759 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294c10c740 con 0x7f294c0f55a0 2021-11-20T05:38:18.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.023+0000 7f294affd700 1 -- 172.21.15.58:0/231976759 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2944018070 con 0x7f294c0f55a0 2021-11-20T05:38:18.027 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.023+0000 7f2954342700 1 -- 172.21.15.58:0/231976759 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294c10cc20 con 0x7f294c0f55a0 2021-11-20T05:38:18.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.024+0000 7f2954342700 1 -- 172.21.15.58:0/231976759 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294c027a00 con 0x7f294c0f55a0 2021-11-20T05:38:18.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.025+0000 7f294affd700 1 -- 172.21.15.58:0/231976759 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f294401dab0 con 0x7f294c0f55a0 2021-11-20T05:38:18.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.025+0000 7f294affd700 1 --2- 172.21.15.58:0/231976759 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f293405c2a0 0x7f293405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:18.028 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.025+0000 7f294affd700 1 -- 172.21.15.58:0/231976759 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f29440780c0 con 0x7f294c0f55a0 2021-11-20T05:38:18.029 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.025+0000 7f29518dd700 1 --2- 172.21.15.58:0/231976759 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f293405c2a0 0x7f293405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:18.029 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.026+0000 7f29518dd700 1 --2- 172.21.15.58:0/231976759 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f293405c2a0 0x7f293405e760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f2940009610 tx=0x7f2940006ab0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:18.029 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.029+0000 7f294affd700 1 -- 172.21.15.58:0/231976759 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f294404a950 con 0x7f294c0f55a0 2021-11-20T05:38:18.223 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.227+0000 7f2954342700 1 -- 172.21.15.58:0/231976759 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f294c0fddb0 con 0x7f294c0f55a0 2021-11-20T05:38:18.223 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.228+0000 7f294affd700 1 -- 172.21.15.58:0/231976759 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f294404df70 con 0x7f294c0f55a0 2021-11-20T05:38:18.224 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:18.224 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:15.310544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24543,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24543":{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:18.226 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.229+0000 7f2954342700 1 -- 172.21.15.58:0/231976759 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f293405c2a0 msgr2=0x7f293405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:18.226 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.229+0000 7f2954342700 1 --2- 172.21.15.58:0/231976759 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f293405c2a0 0x7f293405e760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f2940009610 tx=0x7f2940006ab0).stop 2021-11-20T05:38:18.226 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.229+0000 7f2954342700 1 -- 172.21.15.58:0/231976759 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f294c0f55a0 msgr2=0x7f294c05ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:18.227 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.229+0000 7f2954342700 1 --2- 172.21.15.58:0/231976759 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f294c0f55a0 0x7f294c05ff20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f2944005010 tx=0x7f2944004d10).stop 2021-11-20T05:38:18.227 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.229+0000 7f2954342700 1 -- 172.21.15.58:0/231976759 shutdown_connections 2021-11-20T05:38:18.227 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.229+0000 7f2954342700 1 --2- 172.21.15.58:0/231976759 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f293405c2a0 0x7f293405e760 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:18.227 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.229+0000 7f2954342700 1 --2- 172.21.15.58:0/231976759 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f294c0f55a0 0x7f294c05ff20 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:18.228 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.229+0000 7f2954342700 1 --2- 172.21.15.58:0/231976759 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f294c0f6840 0x7f294c10be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:38:18.228 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.229+0000 7f2954342700 1 -- 172.21.15.58:0/231976759 >> 172.21.15.58:0/231976759 conn(0x7f294c0f0ce0 msgr2=0x7f294c0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:18.228 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.229+0000 7f2954342700 1 -- 172.21.15.58:0/231976759 shutdown_connections 2021-11-20T05:38:18.228 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:18.229+0000 7f2954342700 1 -- 172.21.15.58:0/231976759 wait complete. 2021-11-20T05:38:18.229 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 90 2021-11-20T05:38:18.248 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:15.310544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24543, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24543': {'gid': 24543, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 86, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6825/1821831102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1821831102}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1821831102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:18.249 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24543:up:rejoin 2021-11-20T05:38:18.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:18 smithi187 conmon[36358]: audit 2021-11-20T05:38:18.228727+0000 mon.smithi058 (mon.0) 836 : audit [DBG] from='client.? 172.21.15.58:0/231976759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:18.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:18 smithi058 conmon[33537]: audit 2021-11-20T05:38:18.228727+0000 mon.smithi058 (mon 2021-11-20T05:38:18.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:18 smithi058 conmon[33537]: .0) 836 : audit [DBG] from='client.? 172.21.15.58:0/231976759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:19.250 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:19.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.390+0000 7f0b4cd31700 1 -- 172.21.15.58:0/3539585912 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b480fbab0 msgr2=0x7f0b480fdea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:19.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.390+0000 7f0b4cd31700 1 --2- 172.21.15.58:0/3539585912 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b480fbab0 0x7f0b480fdea0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f0b38004690 tx=0x7f0b38009b30).stop 2021-11-20T05:38:19.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.390+0000 7f0b4cd31700 1 -- 172.21.15.58:0/3539585912 shutdown_connections 2021-11-20T05:38:19.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.390+0000 7f0b4cd31700 1 --2- 172.21.15.58:0/3539585912 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b480fbab0 0x7f0b480fdea0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:19.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.390+0000 7f0b4cd31700 1 --2- 172.21.15.58:0/3539585912 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b480600c0 0x7f0b480fb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:19.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.390+0000 7f0b4cd31700 1 -- 172.21.15.58:0/3539585912 >> 172.21.15.58:0/3539585912 conn(0x7f0b480f0ba0 msgr2=0x7f0b480f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:19.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.390+0000 7f0b4cd31700 1 -- 172.21.15.58:0/3539585912 shutdown_connections 2021-11-20T05:38:19.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.390+0000 7f0b4cd31700 1 -- 172.21.15.58:0/3539585912 wait complete. 2021-11-20T05:38:19.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.390+0000 7f0b4cd31700 1 Processor -- start 2021-11-20T05:38:19.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.391+0000 7f0b4cd31700 1 -- start start 2021-11-20T05:38:19.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.391+0000 7f0b4cd31700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b480600c0 0x7f0b48108ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:19.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.391+0000 7f0b4cd31700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b480fbab0 0x7f0b481093b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:19.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.391+0000 7f0b4cd31700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b48109960 con 0x7f0b480fbab0 2021-11-20T05:38:19.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.391+0000 7f0b4cd31700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b48109aa0 con 0x7f0b480600c0 2021-11-20T05:38:19.391 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.391+0000 7f0b46ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b480fbab0 0x7f0b481093b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:19.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.391+0000 7f0b46ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b480fbab0 0x7f0b481093b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42120/0 (socket says 172.21.15.58:42120) 2021-11-20T05:38:19.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.391+0000 7f0b46ffd700 1 -- 172.21.15.58:0/2543744359 learned_addr learned my addr 172.21.15.58:0/2543744359 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:19.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.391+0000 7f0b477fe700 1 --2- 172.21.15.58:0/2543744359 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b480600c0 0x7f0b48108ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:19.392 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.391+0000 7f0b46ffd700 1 -- 172.21.15.58:0/2543744359 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b480600c0 msgr2=0x7f0b48108ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:19.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.391+0000 7f0b46ffd700 1 --2- 172.21.15.58:0/2543744359 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b480600c0 0x7f0b48108ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:19.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.391+0000 7f0b46ffd700 1 -- 172.21.15.58:0/2543744359 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b38005040 con 0x7f0b480fbab0 2021-11-20T05:38:19.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.392+0000 7f0b46ffd700 1 --2- 172.21.15.58:0/2543744359 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b480fbab0 0x7f0b481093b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f0b38007a40 tx=0x7f0b38009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:19.393 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.392+0000 7f0b3ffff700 1 -- 172.21.15.58:0/2543744359 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b38018070 con 0x7f0b480fbab0 2021-11-20T05:38:19.394 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.392+0000 7f0b4cd31700 1 -- 172.21.15.58:0/2543744359 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b4810a3e0 con 0x7f0b480fbab0 2021-11-20T05:38:19.394 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.392+0000 7f0b3ffff700 1 -- 172.21.15.58:0/2543744359 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b38004ed0 con 0x7f0b480fbab0 2021-11-20T05:38:19.394 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.392+0000 7f0b3ffff700 1 -- 172.21.15.58:0/2543744359 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b3801c620 con 0x7f0b480fbab0 2021-11-20T05:38:19.395 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.392+0000 7f0b4cd31700 1 -- 172.21.15.58:0/2543744359 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b4810a940 con 0x7f0b480fbab0 2021-11-20T05:38:19.395 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.393+0000 7f0b4cd31700 1 -- 172.21.15.58:0/2543744359 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b48045ad0 con 0x7f0b480fbab0 2021-11-20T05:38:19.396 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.393+0000 7f0b3ffff700 1 -- 172.21.15.58:0/2543744359 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0b3801c8e0 con 0x7f0b480fbab0 2021-11-20T05:38:19.396 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.393+0000 7f0b3ffff700 1 --2- 172.21.15.58:0/2543744359 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0b3405c300 0x7f0b3405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:19.397 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.393+0000 7f0b3ffff700 1 -- 172.21.15.58:0/2543744359 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f0b380780c0 con 0x7f0b480fbab0 2021-11-20T05:38:19.397 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.394+0000 7f0b477fe700 1 --2- 172.21.15.58:0/2543744359 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0b3405c300 0x7f0b3405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:19.397 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.394+0000 7f0b477fe700 1 --2- 172.21.15.58:0/2543744359 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0b3405c300 0x7f0b3405e7c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f0b300096c0 tx=0x7f0b30009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:19.397 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.398+0000 7f0b3ffff700 1 -- 172.21.15.58:0/2543744359 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b3804a8d0 con 0x7f0b480fbab0 2021-11-20T05:38:19.591 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.595+0000 7f0b4cd31700 1 -- 172.21.15.58:0/2543744359 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b48027a00 con 0x7f0b480fbab0 2021-11-20T05:38:19.591 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.595+0000 7f0b3ffff700 1 -- 172.21.15.58:0/2543744359 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f0b3804def0 con 0x7f0b480fbab0 2021-11-20T05:38:19.592 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:19.592 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:15.310544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24543,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24543":{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:19.593 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.597+0000 7f0b4cd31700 1 -- 172.21.15.58:0/2543744359 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0b3405c300 msgr2=0x7f0b3405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:19.593 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.597+0000 7f0b4cd31700 1 --2- 172.21.15.58:0/2543744359 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0b3405c300 0x7f0b3405e7c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f0b300096c0 tx=0x7f0b30009be0).stop 2021-11-20T05:38:19.594 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.597+0000 7f0b4cd31700 1 -- 172.21.15.58:0/2543744359 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b480fbab0 msgr2=0x7f0b481093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:19.594 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.597+0000 7f0b4cd31700 1 --2- 172.21.15.58:0/2543744359 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b480fbab0 0x7f0b481093b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f0b38007a40 tx=0x7f0b38009b30).stop 2021-11-20T05:38:19.594 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.597+0000 7f0b4cd31700 1 -- 172.21.15.58:0/2543744359 shutdown_connections 2021-11-20T05:38:19.594 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.597+0000 7f0b4cd31700 1 --2- 172.21.15.58:0/2543744359 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0b3405c300 0x7f0b3405e7c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:19.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.597+0000 7f0b4cd31700 1 --2- 172.21.15.58:0/2543744359 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b480600c0 0x7f0b48108ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:19.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.597+0000 7f0b4cd31700 1 --2- 172.21.15.58:0/2543744359 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0b480fbab0 0x7f0b481093b0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:19.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.597+0000 7f0b4cd31700 1 -- 172.21.15.58:0/2543744359 >> 172.21.15.58:0/2543744359 conn(0x7f0b480f0ba0 msgr2=0x7f0b480f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:19.595 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.597+0000 7f0b4cd31700 1 -- 172.21.15.58:0/2543744359 shutdown_connections 2021-11-20T05:38:19.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:19.597+0000 7f0b4cd31700 1 -- 172.21.15.58:0/2543744359 wait complete. 2021-11-20T05:38:19.596 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 90 2021-11-20T05:38:19.603 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:19 smithi058 conmon[33537]: cluster 2021-11-20T05:38:18.297655+0000 mgr.smithi058.zagdlk 2021-11-20T05:38:19.604 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:19 smithi058 conmon[33537]: (mgr.14164) 318 : cluster [DBG] pgmap v294: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:19.605 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:15.310544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24543, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24543': {'gid': 24543, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 86, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6825/1821831102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1821831102}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1821831102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:19.606 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24543:up:rejoin 2021-11-20T05:38:19.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:19 smithi187 conmon[36358]: cluster 2021-11-20T05:38:18.297655+0000 mgr.smithi058.zagdlk (mgr.14164) 318 : cluster [DBG] pgmap v294: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:20.606 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:20.635 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:20 smithi058 conmon[33537]: audit 2021-11-20T05:38:19.596324+0000 mon.smithi058 2021-11-20T05:38:20.635 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:20 smithi058 conmon[33537]: (mon.0) 837 : audit [DBG] from='client.? 172.21.15.58:0/2543744359' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:20.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:20 smithi187 conmon[36358]: audit 2021-11-20T05:38:19.596324+0000 mon.smithi058 (mon.0) 837 : audit 2021-11-20T05:38:20.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:20 smithi187 conmon[36358]: [DBG] from='client.? 172.21.15.58:0/2543744359' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:20.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.743+0000 7f06ab43b700 1 -- 172.21.15.58:0/862814318 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f06a40f6c60 msgr2=0x7f06a40f7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:20.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.743+0000 7f06ab43b700 1 --2- 172.21.15.58:0/862814318 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f06a40f6c60 0x7f06a40f7080 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f06a0004660 tx=0x7f06a0009b30).stop 2021-11-20T05:38:20.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.743+0000 7f06ab43b700 1 -- 172.21.15.58:0/862814318 shutdown_connections 2021-11-20T05:38:20.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.743+0000 7f06ab43b700 1 --2- 172.21.15.58:0/862814318 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06a40f5890 0x7f06a40f5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:38:20.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.743+0000 7f06ab43b700 1 --2- 172.21.15.58:0/862814318 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f06a40f6c60 0x7f06a40f7080 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:20.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.743+0000 7f06ab43b700 1 -- 172.21.15.58:0/862814318 >> 172.21.15.58:0/862814318 conn(0x7f06a40f04a0 msgr2=0x7f06a40f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:20.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.743+0000 7f06ab43b700 1 -- 172.21.15.58:0/862814318 shutdown_connections 2021-11-20T05:38:20.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.743+0000 7f06ab43b700 1 -- 172.21.15.58:0/862814318 wait complete. 2021-11-20T05:38:20.745 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.744+0000 7f06ab43b700 1 Processor -- start 2021-11-20T05:38:20.745 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.744+0000 7f06ab43b700 1 -- start start 2021-11-20T05:38:20.745 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.744+0000 7f06ab43b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f06a40f5890 0x7f06a40f5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:20.745 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.744+0000 7f06ab43b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06a4109f60 0x7f06a410c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:20.745 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.744+0000 7f06ab43b700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a410c840 con 0x7f06a40f5890 2021-11-20T05:38:20.746 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.744+0000 7f06ab43b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a410c980 con 0x7f06a4109f60 2021-11-20T05:38:20.746 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.744+0000 7f06a91d7700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f06a40f5890 0x7f06a40f5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:20.747 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.744+0000 7f06a91d7700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f06a40f5890 0x7f06a40f5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42132/0 (socket says 172.21.15.58:42132) 2021-11-20T05:38:20.747 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.744+0000 7f06a91d7700 1 -- 172.21.15.58:0/2032442867 learned_addr learned my addr 172.21.15.58:0/2032442867 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:20.747 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.744+0000 7f06a89d6700 1 --2- 172.21.15.58:0/2032442867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06a4109f60 0x7f06a410c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:20.747 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.745+0000 7f06a91d7700 1 -- 172.21.15.58:0/2032442867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06a4109f60 msgr2=0x7f06a410c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:20.748 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.745+0000 7f06a91d7700 1 --2- 172.21.15.58:0/2032442867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06a4109f60 0x7f06a410c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:20.748 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.745+0000 7f06a91d7700 1 -- 172.21.15.58:0/2032442867 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a0005040 con 0x7f06a40f5890 2021-11-20T05:38:20.748 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.745+0000 7f06a91d7700 1 --2- 172.21.15.58:0/2032442867 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f06a40f5890 0x7f06a40f5250 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f06a0009830 tx=0x7f06a0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:20.748 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.745+0000 7f0699ffb700 1 -- 172.21.15.58:0/2032442867 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06a0019070 con 0x7f06a40f5890 2021-11-20T05:38:20.748 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.745+0000 7f0699ffb700 1 -- 172.21.15.58:0/2032442867 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06a0013410 con 0x7f06a40f5890 2021-11-20T05:38:20.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.745+0000 7f06ab43b700 1 -- 172.21.15.58:0/2032442867 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a410cc00 con 0x7f06a40f5890 2021-11-20T05:38:20.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.745+0000 7f0699ffb700 1 -- 172.21.15.58:0/2032442867 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06a00146a0 con 0x7f06a40f5890 2021-11-20T05:38:20.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.745+0000 7f06ab43b700 1 -- 172.21.15.58:0/2032442867 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a410d050 con 0x7f06a40f5890 2021-11-20T05:38:20.750 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.746+0000 7f06ab43b700 1 -- 172.21.15.58:0/2032442867 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a4027a00 con 0x7f06a40f5890 2021-11-20T05:38:20.750 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.747+0000 7f0699ffb700 1 -- 172.21.15.58:0/2032442867 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f06a0014800 con 0x7f06a40f5890 2021-11-20T05:38:20.750 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.747+0000 7f0699ffb700 1 --2- 172.21.15.58:0/2032442867 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f069005c2a0 0x7f069005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:20.751 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.747+0000 7f0699ffb700 1 -- 172.21.15.58:0/2032442867 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f06a00777c0 con 0x7f06a40f5890 2021-11-20T05:38:20.751 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.747+0000 7f06a89d6700 1 --2- 172.21.15.58:0/2032442867 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f069005c2a0 0x7f069005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:20.752 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.747+0000 7f06a89d6700 1 --2- 172.21.15.58:0/2032442867 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f069005c2a0 0x7f069005e760 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f06a40f6a30 tx=0x7f0694008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:20.752 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.751+0000 7f0699ffb700 1 -- 172.21.15.58:0/2032442867 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06a004a050 con 0x7f06a40f5890 2021-11-20T05:38:20.942 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.946+0000 7f06ab43b700 1 -- 172.21.15.58:0/2032442867 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06a40fdee0 con 0x7f06a40f5890 2021-11-20T05:38:20.943 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.947+0000 7f0699ffb700 1 -- 172.21.15.58:0/2032442867 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f06a004d670 con 0x7f06a40f5890 2021-11-20T05:38:20.943 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:20.944 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:15.310544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24543,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24543":{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:20.945 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.948+0000 7f06ab43b700 1 -- 172.21.15.58:0/2032442867 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f069005c2a0 msgr2=0x7f069005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:20.945 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.948+0000 7f06ab43b700 1 --2- 172.21.15.58:0/2032442867 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f069005c2a0 0x7f069005e760 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f06a40f6a30 tx=0x7f0694008040).stop 2021-11-20T05:38:20.945 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.948+0000 7f06ab43b700 1 -- 172.21.15.58:0/2032442867 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f06a40f5890 msgr2=0x7f06a40f5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:20.946 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.948+0000 7f06ab43b700 1 --2- 172.21.15.58:0/2032442867 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f06a40f5890 0x7f06a40f5250 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f06a0009830 tx=0x7f06a0009b30).stop 2021-11-20T05:38:20.946 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.948+0000 7f06ab43b700 1 -- 172.21.15.58:0/2032442867 shutdown_connections 2021-11-20T05:38:20.946 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.948+0000 7f06ab43b700 1 --2- 172.21.15.58:0/2032442867 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f069005c2a0 0x7f069005e760 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:20.946 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.948+0000 7f06ab43b700 1 --2- 172.21.15.58:0/2032442867 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f06a40f5890 0x7f06a40f5250 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:20.947 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.948+0000 7f06ab43b700 1 --2- 172.21.15.58:0/2032442867 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06a4109f60 0x7f06a410c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:20.947 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.948+0000 7f06ab43b700 1 -- 172.21.15.58:0/2032442867 >> 172.21.15.58:0/2032442867 conn(0x7f06a40f04a0 msgr2=0x7f06a40fa2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:20.947 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.949+0000 7f06ab43b700 1 -- 172.21.15.58:0/2032442867 shutdown_connections 2021-11-20T05:38:20.947 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:20.949+0000 7f06ab43b700 1 -- 172.21.15.58:0/2032442867 wait complete. 2021-11-20T05:38:20.948 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 90 2021-11-20T05:38:20.956 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:15.310544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24543, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24543': {'gid': 24543, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 86, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6825/1821831102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1821831102}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1821831102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:20.957 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24543:up:rejoin 2021-11-20T05:38:21.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:21 smithi187 conmon[36358]: cluster 2021-11-20T05:38:20.298195+0000 mgr.smithi058.zagdlk (mgr.14164) 319 : cluster [DBG] pgmap v295: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:21.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:21 smithi187 conmon[36358]: audit 2021-11-20T05: 2021-11-20T05:38:21.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:21 smithi187 conmon[36358]: 38:20.947714+0000 mon.smithi058 (mon.0) 838 : audit [DBG] from='client.? 172.21.15.58:0/2032442867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:21.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:21 smithi058 conmon[33537]: cluster 2021-11-20T05:38:20.298195+0000 2021-11-20T05:38:21.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:21 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 319 : cluster [DBG] pgmap v295: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:21.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:21 smithi058 conmon[33537]: audit 2021-11-20T05:38:20. 2021-11-20T05:38:21.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:21 smithi058 conmon[33537]: 947714+0000 mon.smithi058 (mon.0) 838 : audit [DBG] from='client.? 172.21.15.58:0/2032442867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:21.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:21 smithi058 conmon[33537]: 2021-11-20T05:38:21.958 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:22.093 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.095+0000 7f0e65312700 1 -- 172.21.15.58:0/4288899132 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0e600f7410 msgr2=0x7f0e600f46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:22.093 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.095+0000 7f0e65312700 1 --2- 172.21.15.58:0/4288899132 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0e600f7410 0x7f0e600f46a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f0e50004660 tx=0x7f0e50009b30).stop 2021-11-20T05:38:22.094 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.096+0000 7f0e65312700 1 -- 172.21.15.58:0/4288899132 shutdown_connections 2021-11-20T05:38:22.094 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.096+0000 7f0e65312700 1 --2- 172.21.15.58:0/4288899132 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0e600f7410 0x7f0e600f46a0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:22.094 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.096+0000 7f0e65312700 1 --2- 172.21.15.58:0/4288899132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e600f6a70 0x7f0e600f6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:22.094 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.096+0000 7f0e65312700 1 -- 172.21.15.58:0/4288899132 >> 172.21.15.58:0/4288899132 conn(0x7f0e600f0360 msgr2=0x7f0e600f27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:22.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.096+0000 7f0e65312700 1 -- 172.21.15.58:0/4288899132 shutdown_connections 2021-11-20T05:38:22.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.096+0000 7f0e65312700 1 -- 172.21.15.58:0/4288899132 wait complete. 2021-11-20T05:38:22.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.096+0000 7f0e65312700 1 Processor -- start 2021-11-20T05:38:22.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.096+0000 7f0e65312700 1 -- start start 2021-11-20T05:38:22.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.097+0000 7f0e65312700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0e600f6a70 0x7f0e60108f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:22.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.097+0000 7f0e65312700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e600f7410 0x7f0e60109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:22.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.097+0000 7f0e65312700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e60109a00 con 0x7f0e600f6a70 2021-11-20T05:38:22.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.097+0000 7f0e65312700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e60109b40 con 0x7f0e600f7410 2021-11-20T05:38:22.097 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.097+0000 7f0e5ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0e600f6a70 0x7f0e60108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:22.097 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.097+0000 7f0e5ffff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0e600f6a70 0x7f0e60108f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42142/0 (socket says 172.21.15.58:42142) 2021-11-20T05:38:22.097 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.097+0000 7f0e5ffff700 1 -- 172.21.15.58:0/1225815243 learned_addr learned my addr 172.21.15.58:0/1225815243 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:22.097 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.097+0000 7f0e5ffff700 1 -- 172.21.15.58:0/1225815243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e600f7410 msgr2=0x7f0e60109450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:22.097 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.097+0000 7f0e5ffff700 1 --2- 172.21.15.58:0/1225815243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e600f7410 0x7f0e60109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:38:22.098 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.097+0000 7f0e5ffff700 1 -- 172.21.15.58:0/1225815243 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e50005040 con 0x7f0e600f6a70 2021-11-20T05:38:22.098 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.097+0000 7f0e5ffff700 1 --2- 172.21.15.58:0/1225815243 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0e600f6a70 0x7f0e60108f60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f0e540026d0 tx=0x7f0e54009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:22.098 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.097+0000 7f0e5cff9700 1 -- 172.21.15.58:0/1225815243 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e54009320 con 0x7f0e600f6a70 2021-11-20T05:38:22.098 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.097+0000 7f0e5cff9700 1 -- 172.21.15.58:0/1225815243 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e5400f040 con 0x7f0e600f6a70 2021-11-20T05:38:22.099 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.097+0000 7f0e65312700 1 -- 172.21.15.58:0/1225815243 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e6010a4e0 con 0x7f0e600f6a70 2021-11-20T05:38:22.099 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.097+0000 7f0e5cff9700 1 -- 172.21.15.58:0/1225815243 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e540147e0 con 0x7f0e600f6a70 2021-11-20T05:38:22.099 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.097+0000 7f0e65312700 1 -- 172.21.15.58:0/1225815243 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e6010aaa0 con 0x7f0e600f6a70 2021-11-20T05:38:22.101 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.098+0000 7f0e65312700 1 -- 172.21.15.58:0/1225815243 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e60045ad0 con 0x7f0e600f6a70 2021-11-20T05:38:22.101 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.099+0000 7f0e5cff9700 1 -- 172.21.15.58:0/1225815243 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0e54009480 con 0x7f0e600f6a70 2021-11-20T05:38:22.101 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.099+0000 7f0e5cff9700 1 --2- 172.21.15.58:0/1225815243 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0e4805c300 0x7f0e4805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:22.102 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.099+0000 7f0e5cff9700 1 -- 172.21.15.58:0/1225815243 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f0e54076960 con 0x7f0e600f6a70 2021-11-20T05:38:22.102 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.099+0000 7f0e5f7fe700 1 --2- 172.21.15.58:0/1225815243 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0e4805c300 0x7f0e4805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:22.102 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.100+0000 7f0e5f7fe700 1 --2- 172.21.15.58:0/1225815243 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0e4805c300 0x7f0e4805e7c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f0e50009f50 tx=0x7f0e50009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:22.102 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.103+0000 7f0e5cff9700 1 -- 172.21.15.58:0/1225815243 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e540491f0 con 0x7f0e600f6a70 2021-11-20T05:38:22.295 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.299+0000 7f0e65312700 1 -- 172.21.15.58:0/1225815243 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e60027a00 con 0x7f0e600f6a70 2021-11-20T05:38:22.478 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.300+0000 7f0e5cff9700 1 -- 172.21.15.58:0/1225815243 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f0e5404c810 con 0x7f0e600f6a70 2021-11-20T05:38:22.479 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:22.479 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:15.310544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24543,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24543":{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:22.481 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.302+0000 7f0e65312700 1 -- 172.21.15.58:0/1225815243 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0e4805c300 msgr2=0x7f0e4805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:22.481 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.302+0000 7f0e65312700 1 --2- 172.21.15.58:0/1225815243 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0e4805c300 0x7f0e4805e7c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f0e50009f50 tx=0x7f0e50009b30).stop 2021-11-20T05:38:22.481 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.302+0000 7f0e65312700 1 -- 172.21.15.58:0/1225815243 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0e600f6a70 msgr2=0x7f0e60108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:22.482 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.302+0000 7f0e65312700 1 --2- 172.21.15.58:0/1225815243 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0e600f6a70 0x7f0e60108f60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f0e540026d0 tx=0x7f0e54009de0).stop 2021-11-20T05:38:22.482 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.302+0000 7f0e65312700 1 -- 172.21.15.58:0/1225815243 shutdown_connections 2021-11-20T05:38:22.482 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.302+0000 7f0e65312700 1 --2- 172.21.15.58:0/1225815243 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f0e4805c300 0x7f0e4805e7c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:22.482 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.302+0000 7f0e65312700 1 --2- 172.21.15.58:0/1225815243 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f0e600f6a70 0x7f0e60108f60 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:22.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.302+0000 7f0e65312700 1 --2- 172.21.15.58:0/1225815243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e600f7410 0x7f0e60109450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:38:22.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.302+0000 7f0e65312700 1 -- 172.21.15.58:0/1225815243 >> 172.21.15.58:0/1225815243 conn(0x7f0e600f0360 msgr2=0x7f0e600fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:22.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.302+0000 7f0e65312700 1 -- 172.21.15.58:0/1225815243 shutdown_connections 2021-11-20T05:38:22.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:22.302+0000 7f0e65312700 1 -- 172.21.15.58:0/1225815243 wait complete. 2021-11-20T05:38:22.483 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 90 2021-11-20T05:38:22.484 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:15.310544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24543, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24543': {'gid': 24543, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 86, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6825/1821831102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1821831102}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1821831102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:22.484 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24543:up:rejoin 2021-11-20T05:38:22.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:22 smithi187 conmon[36358]: audit 2021-11-20T05:38:22.300943+0000 mon.smithi058 (mon.0) 839 : audit [DBG] 2021-11-20T05:38:22.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:22 smithi187 conmon[36358]: from='client.? 172.21.15.58:0/1225815243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:22.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:22 smithi058 conmon[33537]: audit 2021-11-20T05:38:22.300943+0000 mon.smithi058 (mon 2021-11-20T05:38:22.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:22 smithi058 conmon[33537]: .0) 839 : audit [DBG] from='client.? 172.21.15.58:0/1225815243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:23.486 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:23.624 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.623+0000 7f4506a2b700 1 -- 172.21.15.58:0/1763635623 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f45000f7960 msgr2=0x7f45000f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:23.624 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.623+0000 7f4506a2b700 1 --2- 172.21.15.58:0/1763635623 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f45000f7960 0x7f45000f7dc0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f44e8004660 tx=0x7f44e8009b30).stop 2021-11-20T05:38:23.624 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.623+0000 7f4506a2b700 1 -- 172.21.15.58:0/1763635623 shutdown_connections 2021-11-20T05:38:23.624 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.623+0000 7f4506a2b700 1 --2- 172.21.15.58:0/1763635623 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f45000f7960 0x7f45000f7dc0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:23.625 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.623+0000 7f4506a2b700 1 --2- 172.21.15.58:0/1763635623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45000f66c0 0x7f45000f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:38:23.625 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.623+0000 7f4506a2b700 1 -- 172.21.15.58:0/1763635623 >> 172.21.15.58:0/1763635623 conn(0x7f45000f1e40 msgr2=0x7f45000f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:23.625 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.623+0000 7f4506a2b700 1 -- 172.21.15.58:0/1763635623 shutdown_connections 2021-11-20T05:38:23.625 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.623+0000 7f4506a2b700 1 -- 172.21.15.58:0/1763635623 wait complete. 2021-11-20T05:38:23.626 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.624+0000 7f4506a2b700 1 Processor -- start 2021-11-20T05:38:23.626 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.624+0000 7f4506a2b700 1 -- start start 2021-11-20T05:38:23.626 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.624+0000 7f4506a2b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45000f66c0 0x7f4500106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:23.626 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.624+0000 7f4506a2b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f45000f7960 0x7f45001073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:23.626 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.624+0000 7f4506a2b700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45001079e0 con 0x7f45000f7960 2021-11-20T05:38:23.627 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.624+0000 7f4506a2b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4500107b20 con 0x7f45000f66c0 2021-11-20T05:38:23.627 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.624+0000 7f44f7fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f45000f7960 0x7f45001073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:23.627 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.624+0000 7f44f7fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f45000f7960 0x7f45001073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42152/0 (socket says 172.21.15.58:42152) 2021-11-20T05:38:23.628 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.624+0000 7f44f7fff700 1 -- 172.21.15.58:0/4161886261 learned_addr learned my addr 172.21.15.58:0/4161886261 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:23.628 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.624+0000 7f44fffff700 1 --2- 172.21.15.58:0/4161886261 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45000f66c0 0x7f4500106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:23.628 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.625+0000 7f44f7fff700 1 -- 172.21.15.58:0/4161886261 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45000f66c0 msgr2=0x7f4500106eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:23.628 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.625+0000 7f44f7fff700 1 --2- 172.21.15.58:0/4161886261 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45000f66c0 0x7f4500106eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:23.628 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.625+0000 7f44f7fff700 1 -- 172.21.15.58:0/4161886261 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e8005040 con 0x7f45000f7960 2021-11-20T05:38:23.629 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.625+0000 7f44f7fff700 1 --2- 172.21.15.58:0/4161886261 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f45000f7960 0x7f45001073a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f44e8009830 tx=0x7f44e8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:23.629 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.625+0000 7f44fd7fa700 1 -- 172.21.15.58:0/4161886261 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44e8018070 con 0x7f45000f7960 2021-11-20T05:38:23.629 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.625+0000 7f44fd7fa700 1 -- 172.21.15.58:0/4161886261 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44e8013410 con 0x7f45000f7960 2021-11-20T05:38:23.630 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.625+0000 7f4506a2b700 1 -- 172.21.15.58:0/4161886261 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4500069d00 con 0x7f45000f7960 2021-11-20T05:38:23.630 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.625+0000 7f44fd7fa700 1 -- 172.21.15.58:0/4161886261 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44e801c620 con 0x7f45000f7960 2021-11-20T05:38:23.630 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.625+0000 7f4506a2b700 1 -- 172.21.15.58:0/4161886261 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f450006a260 con 0x7f45000f7960 2021-11-20T05:38:23.630 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.626+0000 7f4506a2b700 1 -- 172.21.15.58:0/4161886261 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4500027a00 con 0x7f45000f7960 2021-11-20T05:38:23.631 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.626+0000 7f44fd7fa700 1 -- 172.21.15.58:0/4161886261 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f44e801d420 con 0x7f45000f7960 2021-11-20T05:38:23.631 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.627+0000 7f44fd7fa700 1 --2- 172.21.15.58:0/4161886261 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44ec05c2a0 0x7f44ec05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:23.631 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.627+0000 7f44fd7fa700 1 -- 172.21.15.58:0/4161886261 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f44e8078210 con 0x7f45000f7960 2021-11-20T05:38:23.631 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.627+0000 7f44fffff700 1 --2- 172.21.15.58:0/4161886261 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44ec05c2a0 0x7f44ec05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:23.632 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.627+0000 7f44fffff700 1 --2- 172.21.15.58:0/4161886261 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44ec05c2a0 0x7f44ec05e760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f44f0006fd0 tx=0x7f44f0011040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:23.632 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.631+0000 7f44fd7fa700 1 -- 172.21.15.58:0/4161886261 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44e804aa20 con 0x7f45000f7960 2021-11-20T05:38:23.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:23 smithi187 conmon[36358]: cluster 2021-11-20T05:38:22.298681+0000 mgr.smithi058.zagdlk (mgr.14164) 320 : cluster [DBG] pgmap v296: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T05:38:23.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:23 smithi058 conmon[33537]: cluster 2021-11-20T05:38:22.298681+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:38:23.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:23 smithi058 conmon[33537]: 320 : cluster [DBG] pgmap v296: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T05:38:23.826 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.830+0000 7f4506a2b700 1 -- 172.21.15.58:0/4161886261 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45000feef0 con 0x7f45000f7960 2021-11-20T05:38:23.827 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.831+0000 7f44fd7fa700 1 -- 172.21.15.58:0/4161886261 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f44e804e040 con 0x7f45000f7960 2021-11-20T05:38:23.828 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:23.828 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:15.310544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24543,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24543":{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:23.829 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.832+0000 7f4506a2b700 1 -- 172.21.15.58:0/4161886261 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44ec05c2a0 msgr2=0x7f44ec05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:23.829 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.832+0000 7f4506a2b700 1 --2- 172.21.15.58:0/4161886261 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44ec05c2a0 0x7f44ec05e760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f44f0006fd0 tx=0x7f44f0011040).stop 2021-11-20T05:38:23.830 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.832+0000 7f4506a2b700 1 -- 172.21.15.58:0/4161886261 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f45000f7960 msgr2=0x7f45001073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:23.830 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.832+0000 7f4506a2b700 1 --2- 172.21.15.58:0/4161886261 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f45000f7960 0x7f45001073a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f44e8009830 tx=0x7f44e8009b30).stop 2021-11-20T05:38:23.830 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.833+0000 7f4506a2b700 1 -- 172.21.15.58:0/4161886261 shutdown_connections 2021-11-20T05:38:23.830 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.833+0000 7f4506a2b700 1 --2- 172.21.15.58:0/4161886261 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44ec05c2a0 0x7f44ec05e760 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:23.830 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.833+0000 7f4506a2b700 1 --2- 172.21.15.58:0/4161886261 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45000f66c0 0x7f4500106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:23.831 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.833+0000 7f4506a2b700 1 --2- 172.21.15.58:0/4161886261 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f45000f7960 0x7f45001073a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:23.831 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.833+0000 7f4506a2b700 1 -- 172.21.15.58:0/4161886261 >> 172.21.15.58:0/4161886261 conn(0x7f45000f1e40 msgr2=0x7f45000fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:23.831 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.833+0000 7f4506a2b700 1 -- 172.21.15.58:0/4161886261 shutdown_connections 2021-11-20T05:38:23.831 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:23.833+0000 7f4506a2b700 1 -- 172.21.15.58:0/4161886261 wait complete. 2021-11-20T05:38:23.832 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 90 2021-11-20T05:38:23.840 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:15.310544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24543, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24543': {'gid': 24543, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 86, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6825/1821831102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1821831102}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1821831102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:23.841 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24543:up:rejoin 2021-11-20T05:38:24.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:24 smithi187 conmon[36358]: audit 2021-11-20T05:38:23.831950+0000 mon.smithi058 (mon.0) 840 : audit [DBG] 2021-11-20T05:38:24.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:24 smithi187 conmon[36358]: from='client.? 172.21.15.58:0/4161886261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:24.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:24 smithi058 conmon[33537]: audit 2021-11-20T05:38:23.831950+0000 mon.smithi058 (mon.0) 840 : audit [DBG] from='client.? 172.21.15.58:0/4161886261' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:24.842 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:24.976 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.978+0000 7f1a5ac15700 1 -- 172.21.15.58:0/3278590450 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a540f7940 msgr2=0x7f1a540f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:24.976 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.978+0000 7f1a5ac15700 1 --2- 172.21.15.58:0/3278590450 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a540f7940 0x7f1a540f7da0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f1a44004660 tx=0x7f1a44009b30).stop 2021-11-20T05:38:24.976 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.978+0000 7f1a5ac15700 1 -- 172.21.15.58:0/3278590450 shutdown_connections 2021-11-20T05:38:24.976 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.978+0000 7f1a5ac15700 1 --2- 172.21.15.58:0/3278590450 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a540f7940 0x7f1a540f7da0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:24.977 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.978+0000 7f1a5ac15700 1 --2- 172.21.15.58:0/3278590450 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a540f66a0 0x7f1a540f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:24.977 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.978+0000 7f1a5ac15700 1 -- 172.21.15.58:0/3278590450 >> 172.21.15.58:0/3278590450 conn(0x7f1a540f1da0 msgr2=0x7f1a540f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:24.977 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.978+0000 7f1a5ac15700 1 -- 172.21.15.58:0/3278590450 shutdown_connections 2021-11-20T05:38:24.977 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.978+0000 7f1a5ac15700 1 -- 172.21.15.58:0/3278590450 wait complete. 2021-11-20T05:38:24.978 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.979+0000 7f1a5ac15700 1 Processor -- start 2021-11-20T05:38:24.978 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.979+0000 7f1a5ac15700 1 -- start start 2021-11-20T05:38:24.978 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.979+0000 7f1a5ac15700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a540f66a0 0x7f1a54106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:24.978 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.979+0000 7f1a5ac15700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a540f7940 0x7f1a541073e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:24.979 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.979+0000 7f1a5ac15700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a54107990 con 0x7f1a540f66a0 2021-11-20T05:38:24.979 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.979+0000 7f1a5ac15700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a54107ad0 con 0x7f1a540f7940 2021-11-20T05:38:24.980 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.979+0000 7f1a589b1700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a540f66a0 0x7f1a54106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:24.980 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.979+0000 7f1a589b1700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a540f66a0 0x7f1a54106ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42162/0 (socket says 172.21.15.58:42162) 2021-11-20T05:38:24.980 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.979+0000 7f1a589b1700 1 -- 172.21.15.58:0/116219686 learned_addr learned my addr 172.21.15.58:0/116219686 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:24.981 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.979+0000 7f1a53fff700 1 --2- 172.21.15.58:0/116219686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a540f7940 0x7f1a541073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:24.981 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.979+0000 7f1a589b1700 1 -- 172.21.15.58:0/116219686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a540f7940 msgr2=0x7f1a541073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:24.981 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.979+0000 7f1a589b1700 1 --2- 172.21.15.58:0/116219686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a540f7940 0x7f1a541073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:24.982 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.979+0000 7f1a589b1700 1 -- 172.21.15.58:0/116219686 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a44005040 con 0x7f1a540f66a0 2021-11-20T05:38:24.982 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.979+0000 7f1a53fff700 1 --2- 172.21.15.58:0/116219686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a540f7940 0x7f1a541073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:38:24.982 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.980+0000 7f1a589b1700 1 --2- 172.21.15.58:0/116219686 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a540f66a0 0x7f1a54106ef0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f1a4c00d7f0 tx=0x7f1a4c004e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:24.982 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.980+0000 7f1a517fa700 1 -- 172.21.15.58:0/116219686 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a4c00b4f0 con 0x7f1a540f66a0 2021-11-20T05:38:24.983 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.980+0000 7f1a5ac15700 1 -- 172.21.15.58:0/116219686 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5410c570 con 0x7f1a540f66a0 2021-11-20T05:38:24.983 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.980+0000 7f1a517fa700 1 -- 172.21.15.58:0/116219686 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a4c00f040 con 0x7f1a540f66a0 2021-11-20T05:38:24.983 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.980+0000 7f1a517fa700 1 -- 172.21.15.58:0/116219686 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a4c013a00 con 0x7f1a540f66a0 2021-11-20T05:38:24.983 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.980+0000 7f1a5ac15700 1 -- 172.21.15.58:0/116219686 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5410cb30 con 0x7f1a540f66a0 2021-11-20T05:38:24.985 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.981+0000 7f1a5ac15700 1 -- 172.21.15.58:0/116219686 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a54027a00 con 0x7f1a540f66a0 2021-11-20T05:38:24.985 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.981+0000 7f1a517fa700 1 -- 172.21.15.58:0/116219686 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1a4c01a030 con 0x7f1a540f66a0 2021-11-20T05:38:24.985 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.981+0000 7f1a517fa700 1 --2- 172.21.15.58:0/116219686 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1a4005c2a0 0x7f1a4005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:24.985 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.982+0000 7f1a517fa700 1 -- 172.21.15.58:0/116219686 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1a4c076200 con 0x7f1a540f66a0 2021-11-20T05:38:24.986 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.982+0000 7f1a53fff700 1 --2- 172.21.15.58:0/116219686 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1a4005c2a0 0x7f1a4005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:24.986 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.982+0000 7f1a53fff700 1 --2- 172.21.15.58:0/116219686 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1a4005c2a0 0x7f1a4005e760 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f1a44009d60 tx=0x7f1a44009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:24.987 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:24.986+0000 7f1a517fa700 1 -- 172.21.15.58:0/116219686 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a4c048a80 con 0x7f1a540f66a0 2021-11-20T05:38:25.179 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:25.183+0000 7f1a5ac15700 1 -- 172.21.15.58:0/116219686 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a540fee50 con 0x7f1a540f66a0 2021-11-20T05:38:25.180 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:25.184+0000 7f1a517fa700 1 -- 172.21.15.58:0/116219686 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f1a4c018020 con 0x7f1a540f66a0 2021-11-20T05:38:25.180 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:25.180 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:15.310544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24543,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24543":{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:25.182 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:25.185+0000 7f1a5ac15700 1 -- 172.21.15.58:0/116219686 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1a4005c2a0 msgr2=0x7f1a4005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:25.183 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:25.185+0000 7f1a5ac15700 1 --2- 172.21.15.58:0/116219686 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1a4005c2a0 0x7f1a4005e760 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f1a44009d60 tx=0x7f1a44009b30).stop 2021-11-20T05:38:25.183 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:25.185+0000 7f1a5ac15700 1 -- 172.21.15.58:0/116219686 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a540f66a0 msgr2=0x7f1a54106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:25.183 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:25.185+0000 7f1a5ac15700 1 --2- 172.21.15.58:0/116219686 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a540f66a0 0x7f1a54106ef0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f1a4c00d7f0 tx=0x7f1a4c004e30).stop 2021-11-20T05:38:25.184 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:25.186+0000 7f1a5ac15700 1 -- 172.21.15.58:0/116219686 shutdown_connections 2021-11-20T05:38:25.184 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:25.186+0000 7f1a5ac15700 1 --2- 172.21.15.58:0/116219686 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f1a4005c2a0 0x7f1a4005e760 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:25.184 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:25.186+0000 7f1a5ac15700 1 --2- 172.21.15.58:0/116219686 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f1a540f66a0 0x7f1a54106ef0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:25.184 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:25.186+0000 7f1a5ac15700 1 --2- 172.21.15.58:0/116219686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1a540f7940 0x7f1a541073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:25.184 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:25.186+0000 7f1a5ac15700 1 -- 172.21.15.58:0/116219686 >> 172.21.15.58:0/116219686 conn(0x7f1a540f1da0 msgr2=0x7f1a540fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:25.185 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:25.186+0000 7f1a5ac15700 1 -- 172.21.15.58:0/116219686 shutdown_connections 2021-11-20T05:38:25.185 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:25.186+0000 7f1a5ac15700 1 -- 172.21.15.58:0/116219686 wait complete. 2021-11-20T05:38:25.185 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 90 2021-11-20T05:38:25.194 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:15.310544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24543, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24543': {'gid': 24543, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 86, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6825/1821831102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1821831102}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1821831102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:25.194 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24543:up:rejoin 2021-11-20T05:38:25.465 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:25 smithi058 conmon[33537]: cluster 2021-11-20T05:38:24.299235+0000 mgr.smithi058.zagdlk (mgr.14164) 321 : cluster [DBG] pgmap v297: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:38:25.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:25 smithi058 conmon[33537]: audit 2021-11-20T05:38:25.185019+0000 mon.smithi058 (mon.0) 841 : audit [DBG] from='client.? 172.21.15.58:0/116219686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:25.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:25 smithi187 conmon[36358]: cluster 2021-11-20T05:38:24.299235+0000 mgr.smithi058.zagdlk (mgr.14164) 321 : cluster [DBG] pgmap v297: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:38:25.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:25 smithi187 conmon[36358]: 2021-11-20T05:38:25.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:25 smithi187 conmon[36358]: audit 2021-11-20T05:38:25.185019+0000 mon.smithi058 (mon.0) 841 : audit [DBG] from='client.? 172.21.15.58:0/116219686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:26.196 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:26.332 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.333+0000 7faf9138d700 1 -- 172.21.15.58:0/2710052175 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf8c0f6600 msgr2=0x7faf8c0f6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:26.332 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.333+0000 7faf9138d700 1 --2- 172.21.15.58:0/2710052175 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf8c0f6600 0x7faf8c0f6a20 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7faf840099e0 tx=0x7faf84004d10).stop 2021-11-20T05:38:26.333 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.334+0000 7faf9138d700 1 -- 172.21.15.58:0/2710052175 shutdown_connections 2021-11-20T05:38:26.333 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.334+0000 7faf9138d700 1 --2- 172.21.15.58:0/2710052175 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf8c0f78a0 0x7faf8c0f7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:38:26.333 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.334+0000 7faf9138d700 1 --2- 172.21.15.58:0/2710052175 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf8c0f6600 0x7faf8c0f6a20 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:26.334 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.334+0000 7faf9138d700 1 -- 172.21.15.58:0/2710052175 >> 172.21.15.58:0/2710052175 conn(0x7faf8c0f1d40 msgr2=0x7faf8c0f4180 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:26.334 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.334+0000 7faf9138d700 1 -- 172.21.15.58:0/2710052175 shutdown_connections 2021-11-20T05:38:26.334 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.334+0000 7faf9138d700 1 -- 172.21.15.58:0/2710052175 wait complete. 2021-11-20T05:38:26.334 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.334+0000 7faf9138d700 1 Processor -- start 2021-11-20T05:38:26.335 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.334+0000 7faf9138d700 1 -- start start 2021-11-20T05:38:26.335 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.335+0000 7faf9138d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf8c0f6600 0x7faf8c109160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:26.335 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.335+0000 7faf9138d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf8c0f78a0 0x7faf8c109650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:26.335 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.335+0000 7faf9138d700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c109c90 con 0x7faf8c0f6600 2021-11-20T05:38:26.336 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.335+0000 7faf9138d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c10a540 con 0x7faf8c0f78a0 2021-11-20T05:38:26.336 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.335+0000 7faf8affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf8c0f6600 0x7faf8c109160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:26.336 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.335+0000 7faf8affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf8c0f6600 0x7faf8c109160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42172/0 (socket says 172.21.15.58:42172) 2021-11-20T05:38:26.336 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.335+0000 7faf8affd700 1 -- 172.21.15.58:0/4121641439 learned_addr learned my addr 172.21.15.58:0/4121641439 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:26.336 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.335+0000 7faf8a7fc700 1 --2- 172.21.15.58:0/4121641439 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf8c0f78a0 0x7faf8c109650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:26.337 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.335+0000 7faf8affd700 1 -- 172.21.15.58:0/4121641439 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf8c0f78a0 msgr2=0x7faf8c109650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:26.337 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.335+0000 7faf8affd700 1 --2- 172.21.15.58:0/4121641439 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf8c0f78a0 0x7faf8c109650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:26.337 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.335+0000 7faf8affd700 1 -- 172.21.15.58:0/4121641439 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf84009640 con 0x7faf8c0f6600 2021-11-20T05:38:26.337 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.335+0000 7faf8affd700 1 --2- 172.21.15.58:0/4121641439 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf8c0f6600 0x7faf8c109160 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7faf84005010 tx=0x7faf84004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:26.338 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.335+0000 7faf837fe700 1 -- 172.21.15.58:0/4121641439 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf8400e030 con 0x7faf8c0f6600 2021-11-20T05:38:26.338 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.335+0000 7faf9138d700 1 -- 172.21.15.58:0/4121641439 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8c10a7c0 con 0x7faf8c0f6600 2021-11-20T05:38:26.338 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.335+0000 7faf837fe700 1 -- 172.21.15.58:0/4121641439 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf84007d20 con 0x7faf8c0f6600 2021-11-20T05:38:26.338 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.335+0000 7faf837fe700 1 -- 172.21.15.58:0/4121641439 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf84018070 con 0x7faf8c0f6600 2021-11-20T05:38:26.338 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.335+0000 7faf9138d700 1 -- 172.21.15.58:0/4121641439 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8c10aca0 con 0x7faf8c0f6600 2021-11-20T05:38:26.339 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.336+0000 7faf9138d700 1 -- 172.21.15.58:0/4121641439 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf8c0f59f0 con 0x7faf8c0f6600 2021-11-20T05:38:26.339 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.337+0000 7faf837fe700 1 -- 172.21.15.58:0/4121641439 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7faf8401d420 con 0x7faf8c0f6600 2021-11-20T05:38:26.340 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.337+0000 7faf837fe700 1 --2- 172.21.15.58:0/4121641439 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7faf7805c2f0 0x7faf7805e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:26.341 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.337+0000 7faf837fe700 1 -- 172.21.15.58:0/4121641439 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6586+0+0 (secure 0 0 0) 0x7faf84078090 con 0x7faf8c0f6600 2021-11-20T05:38:26.341 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.337+0000 7faf8a7fc700 1 --2- 172.21.15.58:0/4121641439 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7faf7805c2f0 0x7faf7805e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:26.342 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.337+0000 7faf8a7fc700 1 --2- 172.21.15.58:0/4121641439 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7faf7805c2f0 0x7faf7805e7b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7faf74006fd0 tx=0x7faf74008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:26.342 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.341+0000 7faf837fe700 1 -- 172.21.15.58:0/4121641439 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf8404a8a0 con 0x7faf8c0f6600 2021-11-20T05:38:26.533 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.537+0000 7faf9138d700 1 -- 172.21.15.58:0/4121641439 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf8c045ad0 con 0x7faf8c0f6600 2021-11-20T05:38:26.534 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.538+0000 7faf837fe700 1 -- 172.21.15.58:0/4121641439 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2575 (secure 0 0 0) 0x7faf8404dec0 con 0x7faf8c0f6600 2021-11-20T05:38:26.534 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:26.535 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:15.310544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24543,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24543":{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:26.536 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.539+0000 7faf9138d700 1 -- 172.21.15.58:0/4121641439 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7faf7805c2f0 msgr2=0x7faf7805e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:26.536 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.539+0000 7faf9138d700 1 --2- 172.21.15.58:0/4121641439 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7faf7805c2f0 0x7faf7805e7b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7faf74006fd0 tx=0x7faf74008040).stop 2021-11-20T05:38:26.536 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.539+0000 7faf9138d700 1 -- 172.21.15.58:0/4121641439 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf8c0f6600 msgr2=0x7faf8c109160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:26.536 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.539+0000 7faf9138d700 1 --2- 172.21.15.58:0/4121641439 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf8c0f6600 0x7faf8c109160 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7faf84005010 tx=0x7faf84004d10).stop 2021-11-20T05:38:26.537 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.539+0000 7faf9138d700 1 -- 172.21.15.58:0/4121641439 shutdown_connections 2021-11-20T05:38:26.537 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.539+0000 7faf9138d700 1 --2- 172.21.15.58:0/4121641439 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7faf7805c2f0 0x7faf7805e7b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:26.537 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.539+0000 7faf9138d700 1 --2- 172.21.15.58:0/4121641439 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7faf8c0f6600 0x7faf8c109160 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:26.537 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.539+0000 7faf9138d700 1 --2- 172.21.15.58:0/4121641439 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf8c0f78a0 0x7faf8c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:26.538 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.539+0000 7faf9138d700 1 -- 172.21.15.58:0/4121641439 >> 172.21.15.58:0/4121641439 conn(0x7faf8c0f1d40 msgr2=0x7faf8c0f92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:26.538 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.540+0000 7faf9138d700 1 -- 172.21.15.58:0/4121641439 shutdown_connections 2021-11-20T05:38:26.538 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:26.540+0000 7faf9138d700 1 -- 172.21.15.58:0/4121641439 wait complete. 2021-11-20T05:38:26.538 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 90 2021-11-20T05:38:26.548 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:15.310544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24543, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24543': {'gid': 24543, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 86, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6825/1821831102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1821831102}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1821831102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:26.548 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24543:up:rejoin 2021-11-20T05:38:27.549 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:27.684 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.686+0000 7f14f0140700 1 -- 172.21.15.58:0/1617649524 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14e80f7920 msgr2=0x7f14e80f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:27.685 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.686+0000 7f14f0140700 1 --2- 172.21.15.58:0/1617649524 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14e80f7920 0x7f14e80f7d80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f14d8004660 tx=0x7f14d8009b30).stop 2021-11-20T05:38:27.685 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.686+0000 7f14f0140700 1 -- 172.21.15.58:0/1617649524 shutdown_connections 2021-11-20T05:38:27.685 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.686+0000 7f14f0140700 1 --2- 172.21.15.58:0/1617649524 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14e80f7920 0x7f14e80f7d80 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:27.685 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.686+0000 7f14f0140700 1 --2- 172.21.15.58:0/1617649524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e80f6680 0x7f14e80f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:27.686 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.686+0000 7f14f0140700 1 -- 172.21.15.58:0/1617649524 >> 172.21.15.58:0/1617649524 conn(0x7f14e80f1de0 msgr2=0x7f14e80f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:27.686 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.686+0000 7f14f0140700 1 -- 172.21.15.58:0/1617649524 shutdown_connections 2021-11-20T05:38:27.686 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.686+0000 7f14f0140700 1 -- 172.21.15.58:0/1617649524 wait complete. 2021-11-20T05:38:27.686 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.687+0000 7f14f0140700 1 Processor -- start 2021-11-20T05:38:27.686 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.687+0000 7f14f0140700 1 -- start start 2021-11-20T05:38:27.687 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.687+0000 7f14f0140700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e80f6680 0x7f14e8106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:27.687 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.687+0000 7f14f0140700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14e80f7920 0x7f14e8107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:27.687 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.687+0000 7f14f0140700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e8107930 con 0x7f14e80f7920 2021-11-20T05:38:27.687 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.687+0000 7f14f0140700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e8107a70 con 0x7f14e80f6680 2021-11-20T05:38:27.688 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.687+0000 7f14ed6db700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14e80f7920 0x7f14e8107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:27.688 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.687+0000 7f14ed6db700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14e80f7920 0x7f14e8107380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42184/0 (socket says 172.21.15.58:42184) 2021-11-20T05:38:27.688 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.687+0000 7f14ed6db700 1 -- 172.21.15.58:0/1103476258 learned_addr learned my addr 172.21.15.58:0/1103476258 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:27.688 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.687+0000 7f14ededc700 1 --2- 172.21.15.58:0/1103476258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e80f6680 0x7f14e8106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:27.688 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.687+0000 7f14ed6db700 1 -- 172.21.15.58:0/1103476258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e80f6680 msgr2=0x7f14e8106e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:27.689 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.687+0000 7f14ed6db700 1 --2- 172.21.15.58:0/1103476258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e80f6680 0x7f14e8106e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:27.689 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.687+0000 7f14ed6db700 1 -- 172.21.15.58:0/1103476258 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d8005040 con 0x7f14e80f7920 2021-11-20T05:38:27.689 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.688+0000 7f14ed6db700 1 --2- 172.21.15.58:0/1103476258 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14e80f7920 0x7f14e8107380 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f14d8009f50 tx=0x7f14d8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:27.689 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.688+0000 7f14de7fc700 1 -- 172.21.15.58:0/1103476258 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14d8018070 con 0x7f14e80f7920 2021-11-20T05:38:27.689 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.688+0000 7f14f0140700 1 -- 172.21.15.58:0/1103476258 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e810c4b0 con 0x7f14e80f7920 2021-11-20T05:38:27.690 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.688+0000 7f14de7fc700 1 -- 172.21.15.58:0/1103476258 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14d8013410 con 0x7f14e80f7920 2021-11-20T05:38:27.690 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.688+0000 7f14de7fc700 1 -- 172.21.15.58:0/1103476258 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14d801c3c0 con 0x7f14e80f7920 2021-11-20T05:38:27.690 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.688+0000 7f14f0140700 1 -- 172.21.15.58:0/1103476258 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e810ca70 con 0x7f14e80f7920 2021-11-20T05:38:27.690 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.689+0000 7f14f0140700 1 -- 172.21.15.58:0/1103476258 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e8027a00 con 0x7f14e80f7920 2021-11-20T05:38:27.691 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.689+0000 7f14de7fc700 1 -- 172.21.15.58:0/1103476258 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f14d801d420 con 0x7f14e80f7920 2021-11-20T05:38:27.692 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.689+0000 7f14de7fc700 1 --2- 172.21.15.58:0/1103476258 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f14d405c2a0 0x7f14d405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:27.692 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.690+0000 7f14de7fc700 1 -- 172.21.15.58:0/1103476258 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f14d8077ff0 con 0x7f14e80f7920 2021-11-20T05:38:27.693 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.690+0000 7f14ededc700 1 --2- 172.21.15.58:0/1103476258 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f14d405c2a0 0x7f14d405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:27.693 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.690+0000 7f14ededc700 1 --2- 172.21.15.58:0/1103476258 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f14d405c2a0 0x7f14d405e760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f14e80f7780 tx=0x7f14e400b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:27.693 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.694+0000 7f14de7fc700 1 -- 172.21.15.58:0/1103476258 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14d804a800 con 0x7f14e80f7920 2021-11-20T05:38:27.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:27 smithi187 conmon[36358]: cluster 2021-11-20T05:38:26.299642+0000 mgr.smithi058.zagdlk (mgr.14164) 322 : cluster [DBG] pgmap v298: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 170 B/s rd, 0 op/s 2021-11-20T05:38:27.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:27 smithi187 conmon[36358]: audit 2021-11-20T05:38:26.538945+0000 mon.smithi058 (mon.0) 842 : audit [DBG] from='client.? 172.21.15.58:0/4121641439' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:27.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:27 smithi058 conmon[33537]: cluster 2021-11-20T05:38:26.299642+0000 mgr.smithi058.zagdlk (mgr.14164) 322 2021-11-20T05:38:27.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:27 smithi058 conmon[33537]: : cluster [DBG] pgmap v298: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 170 B/s rd, 0 op/s 2021-11-20T05:38:27.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:27 smithi058 conmon[33537]: audit 2021-11-20T05:38:26.538945+0000 mon.smithi058 (mon.0) 842 : audit [DBG] from='client.? 172.21.15.58:0/4121641439' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:27.883 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.888+0000 7f14f0140700 1 -- 172.21.15.58:0/1103476258 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14e810c640 con 0x7f14e80f7920 2021-11-20T05:38:27.884 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.888+0000 7f14de7fc700 1 -- 172.21.15.58:0/1103476258 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f14e810c640 con 0x7f14e80f7920 2021-11-20T05:38:27.884 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:27.885 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:15.310544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24543,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24543":{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:27.885 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.890+0000 7f14f0140700 1 -- 172.21.15.58:0/1103476258 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f14d405c2a0 msgr2=0x7f14d405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:27.886 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.890+0000 7f14f0140700 1 --2- 172.21.15.58:0/1103476258 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f14d405c2a0 0x7f14d405e760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f14e80f7780 tx=0x7f14e400b040).stop 2021-11-20T05:38:27.886 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.890+0000 7f14f0140700 1 -- 172.21.15.58:0/1103476258 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14e80f7920 msgr2=0x7f14e8107380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:27.886 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.890+0000 7f14f0140700 1 --2- 172.21.15.58:0/1103476258 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14e80f7920 0x7f14e8107380 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f14d8009f50 tx=0x7f14d8009b30).stop 2021-11-20T05:38:27.886 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.890+0000 7f14f0140700 1 -- 172.21.15.58:0/1103476258 shutdown_connections 2021-11-20T05:38:27.887 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.890+0000 7f14f0140700 1 --2- 172.21.15.58:0/1103476258 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f14d405c2a0 0x7f14d405e760 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:27.887 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.890+0000 7f14f0140700 1 --2- 172.21.15.58:0/1103476258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e80f6680 0x7f14e8106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:27.887 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.890+0000 7f14f0140700 1 --2- 172.21.15.58:0/1103476258 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14e80f7920 0x7f14e8107380 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:27.887 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.890+0000 7f14f0140700 1 -- 172.21.15.58:0/1103476258 >> 172.21.15.58:0/1103476258 conn(0x7f14e80f1de0 msgr2=0x7f14e80fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:27.888 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.890+0000 7f14f0140700 1 -- 172.21.15.58:0/1103476258 shutdown_connections 2021-11-20T05:38:27.888 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:27.890+0000 7f14f0140700 1 -- 172.21.15.58:0/1103476258 wait complete. 2021-11-20T05:38:27.888 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 90 2021-11-20T05:38:27.897 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:15.310544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24543, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24543': {'gid': 24543, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 86, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6825/1821831102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1821831102}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1821831102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:27.898 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24543:up:rejoin 2021-11-20T05:38:28.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:28 smithi187 conmon[36358]: audit 2021-11-20T05:38:27.889409+0000 mon.smithi058 (mon.0) 843 : audit [DBG] from='client.? 172.21.15.58:0/1103476258' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:28.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:28 smithi187 conmon[36358]: 2021-11-20T05:38:28.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:28 smithi058 conmon[33537]: audit 2021-11-20T05:38:27.889409+0000 mon.smithi058 (mon 2021-11-20T05:38:28.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:28 smithi058 conmon[33537]: .0) 843 : audit [DBG] from='client.? 172.21.15.58:0/1103476258' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:28.899 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:29.033 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.035+0000 7fabed134700 1 -- 172.21.15.58:0/661185311 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fabe80f6820 msgr2=0x7fabe80f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:29.034 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.035+0000 7fabed134700 1 --2- 172.21.15.58:0/661185311 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fabe80f6820 0x7fabe80f6c80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fabdc004660 tx=0x7fabdc009b30).stop 2021-11-20T05:38:29.034 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.035+0000 7fabed134700 1 -- 172.21.15.58:0/661185311 shutdown_connections 2021-11-20T05:38:29.034 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.035+0000 7fabed134700 1 --2- 172.21.15.58:0/661185311 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fabe80f6820 0x7fabe80f6c80 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:29.035 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.035+0000 7fabed134700 1 --2- 172.21.15.58:0/661185311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fabe80f5580 0x7fabe80f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:29.035 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.035+0000 7fabed134700 1 -- 172.21.15.58:0/661185311 >> 172.21.15.58:0/661185311 conn(0x7fabe80f0ce0 msgr2=0x7fabe80f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:29.035 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.035+0000 7fabed134700 1 -- 172.21.15.58:0/661185311 shutdown_connections 2021-11-20T05:38:29.035 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.035+0000 7fabed134700 1 -- 172.21.15.58:0/661185311 wait complete. 2021-11-20T05:38:29.035 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.036+0000 7fabed134700 1 Processor -- start 2021-11-20T05:38:29.036 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.036+0000 7fabed134700 1 -- start start 2021-11-20T05:38:29.036 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.036+0000 7fabed134700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fabe80f5580 0x7fabe8105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:29.036 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.036+0000 7fabed134700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fabe80f6820 0x7fabe8106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:29.036 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.036+0000 7fabed134700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe8106890 con 0x7fabe80f5580 2021-11-20T05:38:29.037 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.036+0000 7fabed134700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe81069d0 con 0x7fabe80f6820 2021-11-20T05:38:29.037 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.036+0000 7fabe6d9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fabe80f5580 0x7fabe8105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:29.037 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.036+0000 7fabe6d9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fabe80f5580 0x7fabe8105da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42194/0 (socket says 172.21.15.58:42194) 2021-11-20T05:38:29.037 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.036+0000 7fabe6d9d700 1 -- 172.21.15.58:0/2644204278 learned_addr learned my addr 172.21.15.58:0/2644204278 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:29.037 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.036+0000 7fabe659c700 1 --2- 172.21.15.58:0/2644204278 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fabe80f6820 0x7fabe8106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:29.038 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.036+0000 7fabe6d9d700 1 -- 172.21.15.58:0/2644204278 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fabe80f6820 msgr2=0x7fabe8106290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:29.038 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.036+0000 7fabe6d9d700 1 --2- 172.21.15.58:0/2644204278 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fabe80f6820 0x7fabe8106290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:29.038 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.036+0000 7fabe6d9d700 1 -- 172.21.15.58:0/2644204278 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabdc005040 con 0x7fabe80f5580 2021-11-20T05:38:29.038 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.037+0000 7fabe6d9d700 1 --2- 172.21.15.58:0/2644204278 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fabe80f5580 0x7fabe8105da0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fabd800f910 tx=0x7fabd8009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:29.039 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.037+0000 7fabcf7fe700 1 -- 172.21.15.58:0/2644204278 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabd8009500 con 0x7fabe80f5580 2021-11-20T05:38:29.039 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.037+0000 7fabcf7fe700 1 -- 172.21.15.58:0/2644204278 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fabd8009660 con 0x7fabe80f5580 2021-11-20T05:38:29.039 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.037+0000 7fabed134700 1 -- 172.21.15.58:0/2644204278 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe805fa80 con 0x7fabe80f5580 2021-11-20T05:38:29.039 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.037+0000 7fabcf7fe700 1 -- 172.21.15.58:0/2644204278 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabd800de20 con 0x7fabe80f5580 2021-11-20T05:38:29.039 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.037+0000 7fabed134700 1 -- 172.21.15.58:0/2644204278 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe8060040 con 0x7fabe80f5580 2021-11-20T05:38:29.040 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.038+0000 7fabed134700 1 -- 172.21.15.58:0/2644204278 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe80fdd10 con 0x7fabe80f5580 2021-11-20T05:38:29.040 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.038+0000 7fabcf7fe700 1 -- 172.21.15.58:0/2644204278 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fabd800e040 con 0x7fabe80f5580 2021-11-20T05:38:29.041 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.039+0000 7fabcf7fe700 1 --2- 172.21.15.58:0/2644204278 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fabd005c2a0 0x7fabd005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:29.041 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.039+0000 7fabcf7fe700 1 -- 172.21.15.58:0/2644204278 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fabd8078a00 con 0x7fabe80f5580 2021-11-20T05:38:29.041 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.039+0000 7fabe659c700 1 --2- 172.21.15.58:0/2644204278 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fabd005c2a0 0x7fabd005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:29.042 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.039+0000 7fabe659c700 1 --2- 172.21.15.58:0/2644204278 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fabd005c2a0 0x7fabd005e760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fabdc004e30 tx=0x7fabdc009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:29.042 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.042+0000 7fabcf7fe700 1 -- 172.21.15.58:0/2644204278 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fabd804b290 con 0x7fabe80f5580 2021-11-20T05:38:29.236 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.240+0000 7fabed134700 1 -- 172.21.15.58:0/2644204278 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabe80ffcc0 con 0x7fabe80f5580 2021-11-20T05:38:29.237 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.240+0000 7fabcf7fe700 1 -- 172.21.15.58:0/2644204278 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2575 (secure 0 0 0) 0x7fabd804e8b0 con 0x7fabe80f5580 2021-11-20T05:38:29.237 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:29.237 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:15.310544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24543,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24543":{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:29.239 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.242+0000 7fabed134700 1 -- 172.21.15.58:0/2644204278 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fabd005c2a0 msgr2=0x7fabd005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:29.239 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.242+0000 7fabed134700 1 --2- 172.21.15.58:0/2644204278 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fabd005c2a0 0x7fabd005e760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fabdc004e30 tx=0x7fabdc009b30).stop 2021-11-20T05:38:29.240 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.242+0000 7fabed134700 1 -- 172.21.15.58:0/2644204278 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fabe80f5580 msgr2=0x7fabe8105da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:29.240 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.242+0000 7fabed134700 1 --2- 172.21.15.58:0/2644204278 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fabe80f5580 0x7fabe8105da0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fabd800f910 tx=0x7fabd8009d80).stop 2021-11-20T05:38:29.240 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.242+0000 7fabed134700 1 -- 172.21.15.58:0/2644204278 shutdown_connections 2021-11-20T05:38:29.241 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.242+0000 7fabed134700 1 --2- 172.21.15.58:0/2644204278 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fabd005c2a0 0x7fabd005e760 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:29.241 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.242+0000 7fabed134700 1 --2- 172.21.15.58:0/2644204278 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fabe80f5580 0x7fabe8105da0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:29.242 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.242+0000 7fabed134700 1 --2- 172.21.15.58:0/2644204278 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fabe80f6820 0x7fabe8106290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:29.242 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.242+0000 7fabed134700 1 -- 172.21.15.58:0/2644204278 >> 172.21.15.58:0/2644204278 conn(0x7fabe80f0ce0 msgr2=0x7fabe80fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:29.242 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.242+0000 7fabed134700 1 -- 172.21.15.58:0/2644204278 shutdown_connections 2021-11-20T05:38:29.242 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:29.242+0000 7fabed134700 1 -- 172.21.15.58:0/2644204278 wait complete. 2021-11-20T05:38:29.243 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 90 2021-11-20T05:38:29.250 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:15.310544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24543, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24543': {'gid': 24543, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 86, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6825/1821831102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1821831102}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1821831102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:29.251 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24543:up:rejoin 2021-11-20T05:38:29.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:29 smithi187 conmon[36358]: cluster 2021-11-20T05:38:28.300182+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:38:29.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:29 smithi187 conmon[36358]: 323 : cluster [DBG] pgmap v299: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 170 B/s rd, 0 op/s 2021-11-20T05:38:29.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:29 smithi187 conmon[36358]: audit 2021-11-20T05:38:29.241579+0000 mon.smithi058 (mon.0) 844 : audit [DBG] from='client.? 172.21.15.58:0/2644204278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:29.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:29 smithi058 conmon[33537]: cluster 2021-11-20T05:38:28.300182+0000 2021-11-20T05:38:29.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:29 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 323 : cluster [DBG] pgmap v299: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 170 B/s rd, 0 op/s 2021-11-20T05:38:29.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:29 smithi058 conmon[33537]: audit 2021-11-20T05:38:29.241579+0000 mon.smithi058 (mon.0) 844 : audit 2021-11-20T05:38:29.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:29 smithi058 conmon[33537]: [DBG] from='client.? 172.21.15.58:0/2644204278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:30.253 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:30.378 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.382+0000 7fb797228700 1 -- 172.21.15.58:0/1875464718 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb7900f6700 msgr2=0x7fb7900f6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:30.378 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.382+0000 7fb797228700 1 --2- 172.21.15.58:0/1875464718 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb7900f6700 0x7fb7900f6b60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fb78c004660 tx=0x7fb78c009b30).stop 2021-11-20T05:38:30.378 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.382+0000 7fb797228700 1 -- 172.21.15.58:0/1875464718 shutdown_connections 2021-11-20T05:38:30.379 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.382+0000 7fb797228700 1 --2- 172.21.15.58:0/1875464718 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb7900f6700 0x7fb7900f6b60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:30.379 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.382+0000 7fb797228700 1 --2- 172.21.15.58:0/1875464718 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7900f5460 0x7fb7900f5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:30.379 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.382+0000 7fb797228700 1 -- 172.21.15.58:0/1875464718 >> 172.21.15.58:0/1875464718 conn(0x7fb7900f0ba0 msgr2=0x7fb7900f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:30.379 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.382+0000 7fb797228700 1 -- 172.21.15.58:0/1875464718 shutdown_connections 2021-11-20T05:38:30.380 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.382+0000 7fb797228700 1 -- 172.21.15.58:0/1875464718 wait complete. 2021-11-20T05:38:30.382 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.383+0000 7fb797228700 1 Processor -- start 2021-11-20T05:38:30.383 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.383+0000 7fb797228700 1 -- start start 2021-11-20T05:38:30.384 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.383+0000 7fb797228700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7900f5460 0x7fb790106d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:30.384 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.383+0000 7fb797228700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb7900f6700 0x7fb790107240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:30.384 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.383+0000 7fb797228700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb790107840 con 0x7fb7900f6700 2021-11-20T05:38:30.384 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.383+0000 7fb797228700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb790108050 con 0x7fb7900f5460 2021-11-20T05:38:30.384 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.383+0000 7fb795a25700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb7900f6700 0x7fb790107240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:30.385 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.383+0000 7fb795a25700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb7900f6700 0x7fb790107240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42206/0 (socket says 172.21.15.58:42206) 2021-11-20T05:38:30.385 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.383+0000 7fb795a25700 1 -- 172.21.15.58:0/2621153862 learned_addr learned my addr 172.21.15.58:0/2621153862 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:30.385 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.383+0000 7fb796226700 1 --2- 172.21.15.58:0/2621153862 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7900f5460 0x7fb790106d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:30.385 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.383+0000 7fb795a25700 1 -- 172.21.15.58:0/2621153862 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7900f5460 msgr2=0x7fb790106d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:30.386 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.383+0000 7fb795a25700 1 --2- 172.21.15.58:0/2621153862 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7900f5460 0x7fb790106d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:30.386 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.383+0000 7fb795a25700 1 -- 172.21.15.58:0/2621153862 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c005040 con 0x7fb7900f6700 2021-11-20T05:38:30.386 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.384+0000 7fb795a25700 1 --2- 172.21.15.58:0/2621153862 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb7900f6700 0x7fb790107240 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fb78c000c00 tx=0x7fb78c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:30.386 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.384+0000 7fb786ffd700 1 -- 172.21.15.58:0/2621153862 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb78c019070 con 0x7fb7900f6700 2021-11-20T05:38:30.387 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.384+0000 7fb786ffd700 1 -- 172.21.15.58:0/2621153862 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb78c007e70 con 0x7fb7900f6700 2021-11-20T05:38:30.387 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.384+0000 7fb797228700 1 -- 172.21.15.58:0/2621153862 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb790108270 con 0x7fb7900f6700 2021-11-20T05:38:30.387 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.384+0000 7fb786ffd700 1 -- 172.21.15.58:0/2621153862 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb78c014d00 con 0x7fb7900f6700 2021-11-20T05:38:30.387 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.384+0000 7fb797228700 1 -- 172.21.15.58:0/2621153862 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7901087d0 con 0x7fb7900f6700 2021-11-20T05:38:30.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.385+0000 7fb797228700 1 -- 172.21.15.58:0/2621153862 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7900f4850 con 0x7fb7900f6700 2021-11-20T05:38:30.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.385+0000 7fb786ffd700 1 -- 172.21.15.58:0/2621153862 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb78c025070 con 0x7fb7900f6700 2021-11-20T05:38:30.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.385+0000 7fb786ffd700 1 --2- 172.21.15.58:0/2621153862 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb77c05c300 0x7fb77c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:30.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.385+0000 7fb786ffd700 1 -- 172.21.15.58:0/2621153862 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb78c0773d0 con 0x7fb7900f6700 2021-11-20T05:38:30.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.385+0000 7fb796226700 1 --2- 172.21.15.58:0/2621153862 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb77c05c300 0x7fb77c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:30.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.386+0000 7fb796226700 1 --2- 172.21.15.58:0/2621153862 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb77c05c300 0x7fb77c05e7c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fb790106ac0 tx=0x7fb78000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:30.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.389+0000 7fb786ffd700 1 -- 172.21.15.58:0/2621153862 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb78c049be0 con 0x7fb7900f6700 2021-11-20T05:38:30.582 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.587+0000 7fb797228700 1 -- 172.21.15.58:0/2621153862 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb790045ad0 con 0x7fb7900f6700 2021-11-20T05:38:30.583 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.587+0000 7fb786ffd700 1 -- 172.21.15.58:0/2621153862 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2575 (secure 0 0 0) 0x7fb78c04d200 con 0x7fb7900f6700 2021-11-20T05:38:30.584 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:30.584 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:15.310544+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24543,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24543":{"gid":24543,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6825/1821831102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1821831102},{"type":"v1","addr":"172.21.15.187:6825","nonce":1821831102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:30.585 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.589+0000 7fb797228700 1 -- 172.21.15.58:0/2621153862 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb77c05c300 msgr2=0x7fb77c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:30.585 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.589+0000 7fb797228700 1 --2- 172.21.15.58:0/2621153862 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb77c05c300 0x7fb77c05e7c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fb790106ac0 tx=0x7fb78000b040).stop 2021-11-20T05:38:30.586 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.589+0000 7fb797228700 1 -- 172.21.15.58:0/2621153862 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb7900f6700 msgr2=0x7fb790107240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:30.586 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.589+0000 7fb797228700 1 --2- 172.21.15.58:0/2621153862 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb7900f6700 0x7fb790107240 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fb78c000c00 tx=0x7fb78c00b040).stop 2021-11-20T05:38:30.587 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.589+0000 7fb797228700 1 -- 172.21.15.58:0/2621153862 shutdown_connections 2021-11-20T05:38:30.587 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.589+0000 7fb797228700 1 --2- 172.21.15.58:0/2621153862 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fb77c05c300 0x7fb77c05e7c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:30.587 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.589+0000 7fb797228700 1 --2- 172.21.15.58:0/2621153862 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7900f5460 0x7fb790106d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:30.587 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.589+0000 7fb797228700 1 --2- 172.21.15.58:0/2621153862 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fb7900f6700 0x7fb790107240 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:30.588 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.589+0000 7fb797228700 1 -- 172.21.15.58:0/2621153862 >> 172.21.15.58:0/2621153862 conn(0x7fb7900f0ba0 msgr2=0x7fb79005f450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:30.588 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.589+0000 7fb797228700 1 -- 172.21.15.58:0/2621153862 shutdown_connections 2021-11-20T05:38:30.588 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:30.589+0000 7fb797228700 1 -- 172.21.15.58:0/2621153862 wait complete. 2021-11-20T05:38:30.588 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 90 2021-11-20T05:38:30.595 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:30 smithi058 conmon[33537]: audit 2021-11-20T05:38:29.706312+0000 mon.smithi058 2021-11-20T05:38:30.596 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:30 smithi058 conmon[33537]: (mon.0) 845 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:38:30.596 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:30 smithi058 conmon[33537]: audit 2021-11-20T05:38:30.091095+0000 mon.smithi058 (mon.0) 846 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:38:30.599 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:15.310544+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24543, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24543': {'gid': 24543, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 86, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6825/1821831102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1821831102}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1821831102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:30.599 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24543:up:rejoin 2021-11-20T05:38:30.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:30 smithi187 conmon[36358]: audit 2021-11-20T05:38:29.706312+0000 mon.smithi058 (mon.0) 845 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:38:30.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:30 smithi187 conmon[36358]: audit 2021-11-20T05:38:30.091095+0000 mon.smithi058 (mon.0) 846 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' 2021-11-20T05:38:31.601 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:31.630 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:31 smithi058 conmon[33537]: cluster 2021-11-20T05:38:30.300621+0000 mgr.smithi058.zagdlk (mgr.14164) 324 2021-11-20T05:38:31.630 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:31 smithi058 conmon[33537]: : cluster [DBG] pgmap v300: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:38:31.631 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:31 smithi058 conmon[33537]: cluster 2021-11-20T05:38:30.529657+0000 mon.smithi058 (mon.0) 847 : cluster [INF] daemon mds.cephfs.smithi187.wxduhu restarted 2021-11-20T05:38:31.631 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:31 smithi058 conmon[33537]: cluster 2021-11-20T05:38:30.533547+0000 mon.smithi058 (mon.0) 848 : 2021-11-20T05:38:31.631 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:31 smithi058 conmon[33537]: cluster [DBG] osdmap e57: 6 total, 6 up, 6 in 2021-11-20T05:38:31.632 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:31 smithi058 conmon[33537]: audit 2021-11-20T05 2021-11-20T05:38:31.632 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:31 smithi058 conmon[33537]: :38:30.588418+0000 mon.smithi058 (mon.0) 849 : audit 2021-11-20T05:38:31.632 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:31 smithi058 conmon[33537]: [DBG] from='client.? 172.21.15.58:0/2621153862' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:31.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:31 smithi187 conmon[36358]: cluster 2021-11-20T05:38:30.300621+0000 mgr.smithi058.zagdlk (mgr. 2021-11-20T05:38:31.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:31 smithi187 conmon[36358]: 14164) 324 : cluster [DBG] pgmap v300: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:38:31.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:31 smithi187 conmon[36358]: cluster 2021-11-20T05:38:30.529657+0000 mon.smithi058 (mon.0) 847 : cluster [INF] daemon mds.cephfs.smithi187.wxduhu restarted 2021-11-20T05:38:31.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:31 smithi187 conmon[36358]: cluster 2021-11-20T05:38:30.533547+0000 mon.smithi058 (mon.0) 848 : cluster [DBG] osdmap e57: 6 total, 6 up, 6 in 2021-11-20T05:38:31.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:31 smithi187 conmon[36358]: audit 2021-11-20T05:38:30.588418+0000 mon.smithi058 (mon.0) 849 : audit [DBG] from='client.? 172.21.15.58:0/2621153862' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:31.734 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.738+0000 7f1235d3b700 1 -- 172.21.15.58:0/2222324225 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f12300f5580 msgr2=0x7f12300f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:31.735 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.738+0000 7f1235d3b700 1 --2- 172.21.15.58:0/2222324225 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f12300f5580 0x7f12300f59a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f1220004660 tx=0x7f1220009b30).stop 2021-11-20T05:38:31.735 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.738+0000 7f1235d3b700 1 -- 172.21.15.58:0/2222324225 shutdown_connections 2021-11-20T05:38:31.735 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.738+0000 7f1235d3b700 1 --2- 172.21.15.58:0/2222324225 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12300f6820 0x7f12300f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:31.735 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.738+0000 7f1235d3b700 1 --2- 172.21.15.58:0/2222324225 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f12300f5580 0x7f12300f59a0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:31.736 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.738+0000 7f1235d3b700 1 -- 172.21.15.58:0/2222324225 >> 172.21.15.58:0/2222324225 conn(0x7f12300f0ce0 msgr2=0x7f12300f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:31.736 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.739+0000 7f1235d3b700 1 -- 172.21.15.58:0/2222324225 shutdown_connections 2021-11-20T05:38:31.736 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.739+0000 7f1235d3b700 1 -- 172.21.15.58:0/2222324225 wait complete. 2021-11-20T05:38:31.736 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.739+0000 7f1235d3b700 1 Processor -- start 2021-11-20T05:38:31.736 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.739+0000 7f1235d3b700 1 -- start start 2021-11-20T05:38:31.739 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.739+0000 7f1235d3b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12300f5580 0x7f12300604d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:31.739 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.739+0000 7f1235d3b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f12300f6820 0x7f123010bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:31.739 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.739+0000 7f1235d3b700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123010c1b0 con 0x7f12300f6820 2021-11-20T05:38:31.739 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.739+0000 7f1235d3b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123010c2f0 con 0x7f12300f5580 2021-11-20T05:38:31.740 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.740+0000 7f122effd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f12300f6820 0x7f123010bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:31.740 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.740+0000 7f122effd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f12300f6820 0x7f123010bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42216/0 (socket says 172.21.15.58:42216) 2021-11-20T05:38:31.740 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.740+0000 7f122effd700 1 -- 172.21.15.58:0/1640066047 learned_addr learned my addr 172.21.15.58:0/1640066047 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:31.740 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.740+0000 7f122f7fe700 1 --2- 172.21.15.58:0/1640066047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12300f5580 0x7f12300604d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:31.741 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.740+0000 7f122effd700 1 -- 172.21.15.58:0/1640066047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12300f5580 msgr2=0x7f12300604d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:31.741 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.740+0000 7f122effd700 1 --2- 172.21.15.58:0/1640066047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12300f5580 0x7f12300604d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:31.741 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.740+0000 7f122effd700 1 -- 172.21.15.58:0/1640066047 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1220005040 con 0x7f12300f6820 2021-11-20T05:38:31.741 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.740+0000 7f122effd700 1 --2- 172.21.15.58:0/1640066047 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f12300f6820 0x7f123010bcc0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f122400c530 tx=0x7f1224007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:31.741 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.740+0000 7f1217fff700 1 -- 172.21.15.58:0/1640066047 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f122400f650 con 0x7f12300f6820 2021-11-20T05:38:31.742 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.740+0000 7f1235d3b700 1 -- 172.21.15.58:0/1640066047 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123010c550 con 0x7f12300f6820 2021-11-20T05:38:31.742 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.741+0000 7f1235d3b700 1 -- 172.21.15.58:0/1640066047 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123010cae0 con 0x7f12300f6820 2021-11-20T05:38:31.742 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.741+0000 7f1217fff700 1 -- 172.21.15.58:0/1640066047 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f122400fc60 con 0x7f12300f6820 2021-11-20T05:38:31.742 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.741+0000 7f1217fff700 1 -- 172.21.15.58:0/1640066047 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1224012c70 con 0x7f12300f6820 2021-11-20T05:38:31.742 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.742+0000 7f1235d3b700 1 -- 172.21.15.58:0/1640066047 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1210004fa0 con 0x7f12300f6820 2021-11-20T05:38:31.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.742+0000 7f1217fff700 1 -- 172.21.15.58:0/1640066047 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1224010070 con 0x7f12300f6820 2021-11-20T05:38:31.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.742+0000 7f1217fff700 1 --2- 172.21.15.58:0/1640066047 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f121805c2a0 0x7f121805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:31.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.742+0000 7f1217fff700 1 -- 172.21.15.58:0/1640066047 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f1224075f60 con 0x7f12300f6820 2021-11-20T05:38:31.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.742+0000 7f122f7fe700 1 --2- 172.21.15.58:0/1640066047 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f121805c2a0 0x7f121805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:31.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.743+0000 7f122f7fe700 1 --2- 172.21.15.58:0/1640066047 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f121805c2a0 0x7f121805e760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f1220004630 tx=0x7f1220009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:31.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.746+0000 7f1217fff700 1 -- 172.21.15.58:0/1640066047 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12240452b0 con 0x7f12300f6820 2021-11-20T05:38:31.938 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.943+0000 7f1235d3b700 1 -- 172.21.15.58:0/1640066047 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1210005bf0 con 0x7f12300f6820 2021-11-20T05:38:31.939 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.943+0000 7f1217fff700 1 -- 172.21.15.58:0/1640066047 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+2576 (secure 0 0 0) 0x7f1224048720 con 0x7f12300f6820 2021-11-20T05:38:31.939 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:31.939 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:31.359025+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24583,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24583":{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":92,"state":"up:replay","state_seq":2,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:31.941 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.945+0000 7f1235d3b700 1 -- 172.21.15.58:0/1640066047 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f121805c2a0 msgr2=0x7f121805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:31.941 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.945+0000 7f1235d3b700 1 --2- 172.21.15.58:0/1640066047 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f121805c2a0 0x7f121805e760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f1220004630 tx=0x7f1220009b30).stop 2021-11-20T05:38:31.941 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.945+0000 7f1235d3b700 1 -- 172.21.15.58:0/1640066047 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f12300f6820 msgr2=0x7f123010bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:31.942 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.945+0000 7f1235d3b700 1 --2- 172.21.15.58:0/1640066047 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f12300f6820 0x7f123010bcc0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f122400c530 tx=0x7f1224007e20).stop 2021-11-20T05:38:31.942 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.945+0000 7f1235d3b700 1 -- 172.21.15.58:0/1640066047 shutdown_connections 2021-11-20T05:38:31.942 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.945+0000 7f1235d3b700 1 --2- 172.21.15.58:0/1640066047 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f121805c2a0 0x7f121805e760 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:31.942 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.945+0000 7f1235d3b700 1 --2- 172.21.15.58:0/1640066047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f12300f5580 0x7f12300604d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:31.942 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.945+0000 7f1235d3b700 1 --2- 172.21.15.58:0/1640066047 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f12300f6820 0x7f123010bcc0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:31.943 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.945+0000 7f1235d3b700 1 -- 172.21.15.58:0/1640066047 >> 172.21.15.58:0/1640066047 conn(0x7f12300f0ce0 msgr2=0x7f12300fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:31.943 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.945+0000 7f1235d3b700 1 -- 172.21.15.58:0/1640066047 shutdown_connections 2021-11-20T05:38:31.943 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:31.945+0000 7f1235d3b700 1 -- 172.21.15.58:0/1640066047 wait complete. 2021-11-20T05:38:31.943 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 92 2021-11-20T05:38:31.952 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:31.359025+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24583, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 92, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.187:6827/1745887865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 1745887865}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 1745887865}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:31.953 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:replay 2021-11-20T05:38:32.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:32 smithi187 conmon[36358]: cluster 2021-11-20T05:38:31.358873+0000 mon.smithi058 (mon.0) 850 : 2021-11-20T05:38:32.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:32 smithi187 conmon[36358]: cluster [DBG] mds.? [v2:172.21.15.187:6824/1414418853,v1:172.21.15.187:6825/1414418853] up:boot 2021-11-20T05:38:32.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:32 smithi187 conmon[36358]: cluster 2021-11-20T05:38:31.358975+0000 mon.smithi058 (mon.0) 851 : cluster 2021-11-20T05:38:32.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:32 smithi187 conmon[36358]: [INF] Standby daemon mds.cephfs.smithi187.fikisa assigned to filesystem cephfs as rank 0 2021-11-20T05:38:32.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:32 smithi187 conmon[36358]: cluster 2021-11-20T05:38:32.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:32 smithi187 conmon[36358]: 2021-11-20T05:38:31.359328+0000 2021-11-20T05:38:32.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:32 smithi187 conmon[36358]: mon.smithi058 (mon.0) 852 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby, 1 failed 2021-11-20T05:38:32.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:32 smithi187 conmon[36358]: audit 2021-11-20T05:38:32.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:32 smithi187 conmon[36358]: 2021-11-20T05:38:31.359459+0000 mon.smithi058 2021-11-20T05:38:32.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:32 smithi187 conmon[36358]: (mon.0) 853 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.wxduhu"}]: dispatch 2021-11-20T05:38:32.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:32 smithi187 conmon[36358]: cluster 2021-11-20T05: 2021-11-20T05:38:32.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:32 smithi187 conmon[36358]: 38:31.362831+0000 mon.smithi058 (mon.0) 854 : cluster [DBG] 2021-11-20T05:38:32.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:32 smithi187 conmon[36358]: fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:32.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:32 smithi187 conmon[36358]: audit 2021-11-20T05:38:32.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:32 smithi187 conmon[36358]: 2021-11-20T05:38:31.944197+0000 mon.smithi058 ( 2021-11-20T05:38:32.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:32 smithi187 conmon[36358]: mon.0) 855 : audit [DBG] from='client.? 172.21.15.58:0/1640066047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:32.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: cluster 2021-11-20T05:38:31.358873+0000 mon.smithi058 ( 2021-11-20T05:38:32.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: mon.0) 850 : cluster [DBG] mds.? [v2:172.21.15.187:6824/1414418853,v1:172.21.15.187:6825/1414418853] up:boot 2021-11-20T05:38:32.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: cluster 2021-11-20T05:38:31 2021-11-20T05:38:32.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: .358975+0000 mon.smithi058 (mon.0) 851 : cluster [INF] Standby daemon mds.cephfs.smithi187.fikisa assigned to filesystem cephfs as rank 0 2021-11-20T05:38:32.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: cluster 2021-11-20T05:38:32.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: 2021-11-20T05:38:31.359328+0000 mon.smithi058 ( 2021-11-20T05:38:32.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: mon.0) 852 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi058.nuxpgf=up:active} 2 up:standby, 1 failed 2021-11-20T05:38:32.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: audit 2021 2021-11-20T05:38:32.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: -11-20T05:38:31 2021-11-20T05:38:32.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: .359459+0000 mon.smithi058 (mon.0 2021-11-20T05:38:32.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: ) 853 : audit [DBG] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.wxduhu"}]: dispatch 2021-11-20T05:38:32.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: 2021-11-20T05:38:32.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: cluster 2021-11-20T05:38:32.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: 2021-11-20T05 2021-11-20T05:38:32.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: :38:31.362831+0000 2021-11-20T05:38:32.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: mon.smithi058 (mon.0) 854 : 2021-11-20T05:38:32.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:32.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: audit 2021 2021-11-20T05:38:32.719 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: -11-20T05:38: 2021-11-20T05:38:32.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: 31.944197+0000 mon.smithi058 (mon 2021-11-20T05:38:32.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: .0) 855 : audit [DBG] 2021-11-20T05:38:32.720 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:32 smithi058 conmon[33537]: from='client.? 172.21.15.58:0/1640066047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:32.954 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:33.089 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.091+0000 7f169171f700 1 -- 172.21.15.58:0/3755610560 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f168c0f5580 msgr2=0x7f168c0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:33.090 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.091+0000 7f169171f700 1 --2- 172.21.15.58:0/3755610560 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f168c0f5580 0x7f168c0f59a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f1674004660 tx=0x7f1674009b30).stop 2021-11-20T05:38:33.090 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.091+0000 7f169171f700 1 -- 172.21.15.58:0/3755610560 shutdown_connections 2021-11-20T05:38:33.090 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.091+0000 7f169171f700 1 --2- 172.21.15.58:0/3755610560 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f168c0f6820 0x7f168c0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:33.090 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.091+0000 7f169171f700 1 --2- 172.21.15.58:0/3755610560 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f168c0f5580 0x7f168c0f59a0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:33.091 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.091+0000 7f169171f700 1 -- 172.21.15.58:0/3755610560 >> 172.21.15.58:0/3755610560 conn(0x7f168c0f0ce0 msgr2=0x7f168c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:33.091 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.091+0000 7f169171f700 1 -- 172.21.15.58:0/3755610560 shutdown_connections 2021-11-20T05:38:33.091 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.091+0000 7f169171f700 1 -- 172.21.15.58:0/3755610560 wait complete. 2021-11-20T05:38:33.091 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.092+0000 7f169171f700 1 Processor -- start 2021-11-20T05:38:33.092 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.092+0000 7f169171f700 1 -- start start 2021-11-20T05:38:33.092 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.092+0000 7f169171f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f168c0f5580 0x7f168c106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:33.092 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.092+0000 7f169171f700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f168c0f6820 0x7f168c1073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:33.092 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.092+0000 7f169171f700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c107970 con 0x7f168c0f6820 2021-11-20T05:38:33.092 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.092+0000 7f169171f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c107ab0 con 0x7f168c0f5580 2021-11-20T05:38:33.093 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.092+0000 7f168a7fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f168c0f6820 0x7f168c1073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:33.093 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.092+0000 7f168a7fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f168c0f6820 0x7f168c1073c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42224/0 (socket says 172.21.15.58:42224) 2021-11-20T05:38:33.093 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.092+0000 7f168a7fc700 1 -- 172.21.15.58:0/296747913 learned_addr learned my addr 172.21.15.58:0/296747913 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:33.094 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.092+0000 7f168affd700 1 --2- 172.21.15.58:0/296747913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f168c0f5580 0x7f168c106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:33.094 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.092+0000 7f168a7fc700 1 -- 172.21.15.58:0/296747913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f168c0f5580 msgr2=0x7f168c106ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:33.094 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.092+0000 7f168a7fc700 1 --2- 172.21.15.58:0/296747913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f168c0f5580 0x7f168c106ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:33.094 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.092+0000 7f168a7fc700 1 -- 172.21.15.58:0/296747913 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1674005040 con 0x7f168c0f6820 2021-11-20T05:38:33.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.092+0000 7f168a7fc700 1 --2- 172.21.15.58:0/296747913 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f168c0f6820 0x7f168c1073c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f167c0026d0 tx=0x7f167c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:33.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.093+0000 7f16837fe700 1 -- 172.21.15.58:0/296747913 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f167c009320 con 0x7f168c0f6820 2021-11-20T05:38:33.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.093+0000 7f16837fe700 1 -- 172.21.15.58:0/296747913 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f167c00f040 con 0x7f168c0f6820 2021-11-20T05:38:33.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.093+0000 7f16837fe700 1 -- 172.21.15.58:0/296747913 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f167c0147e0 con 0x7f168c0f6820 2021-11-20T05:38:33.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.093+0000 7f169171f700 1 -- 172.21.15.58:0/296747913 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168c10c550 con 0x7f168c0f6820 2021-11-20T05:38:33.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.093+0000 7f169171f700 1 -- 172.21.15.58:0/296747913 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f168c10cb10 con 0x7f168c0f6820 2021-11-20T05:38:33.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.094+0000 7f169171f700 1 -- 172.21.15.58:0/296747913 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f168c0fdd10 con 0x7f168c0f6820 2021-11-20T05:38:33.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.094+0000 7f16837fe700 1 -- 172.21.15.58:0/296747913 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f167c009480 con 0x7f168c0f6820 2021-11-20T05:38:33.097 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.094+0000 7f16837fe700 1 --2- 172.21.15.58:0/296747913 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f167805c2a0 0x7f167805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:33.097 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.095+0000 7f16837fe700 1 -- 172.21.15.58:0/296747913 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f167c076ba0 con 0x7f168c0f6820 2021-11-20T05:38:33.098 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.095+0000 7f168affd700 1 --2- 172.21.15.58:0/296747913 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f167805c2a0 0x7f167805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:33.098 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.095+0000 7f168affd700 1 --2- 172.21.15.58:0/296747913 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f167805c2a0 0x7f167805e760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f1674004e30 tx=0x7f1674009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:33.098 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.099+0000 7f16837fe700 1 -- 172.21.15.58:0/296747913 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f167c0492c0 con 0x7f168c0f6820 2021-11-20T05:38:33.291 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.296+0000 7f169171f700 1 -- 172.21.15.58:0/296747913 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f168c10d210 con 0x7f168c0f6820 2021-11-20T05:38:33.292 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.296+0000 7f16837fe700 1 -- 172.21.15.58:0/296747913 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2577 (secure 0 0 0) 0x7f167c04c8e0 con 0x7f168c0f6820 2021-11-20T05:38:33.292 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:33.293 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:32.360507+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24583,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24583":{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":92,"state":"up:resolve","state_seq":6,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:33.294 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.297+0000 7f169171f700 1 -- 172.21.15.58:0/296747913 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f167805c2a0 msgr2=0x7f167805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:33.294 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.297+0000 7f169171f700 1 --2- 172.21.15.58:0/296747913 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f167805c2a0 0x7f167805e760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f1674004e30 tx=0x7f1674009b30).stop 2021-11-20T05:38:33.294 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.298+0000 7f169171f700 1 -- 172.21.15.58:0/296747913 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f168c0f6820 msgr2=0x7f168c1073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:33.295 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.298+0000 7f169171f700 1 --2- 172.21.15.58:0/296747913 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f168c0f6820 0x7f168c1073c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f167c0026d0 tx=0x7f167c009de0).stop 2021-11-20T05:38:33.295 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.298+0000 7f169171f700 1 -- 172.21.15.58:0/296747913 shutdown_connections 2021-11-20T05:38:33.295 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.298+0000 7f169171f700 1 --2- 172.21.15.58:0/296747913 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f167805c2a0 0x7f167805e760 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:33.295 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.298+0000 7f169171f700 1 --2- 172.21.15.58:0/296747913 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f168c0f5580 0x7f168c106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:33.296 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.298+0000 7f169171f700 1 --2- 172.21.15.58:0/296747913 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f168c0f6820 0x7f168c1073c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:33.296 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.298+0000 7f169171f700 1 -- 172.21.15.58:0/296747913 >> 172.21.15.58:0/296747913 conn(0x7f168c0f0ce0 msgr2=0x7f168c05faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:33.296 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.298+0000 7f169171f700 1 -- 172.21.15.58:0/296747913 shutdown_connections 2021-11-20T05:38:33.296 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:33.298+0000 7f169171f700 1 -- 172.21.15.58:0/296747913 wait complete. 2021-11-20T05:38:33.297 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 93 2021-11-20T05:38:33.305 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 93, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:32.360507+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24583, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 92, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.187:6827/1745887865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 1745887865}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 1745887865}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:33.305 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:resolve 2021-11-20T05:38:33.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:33 smithi187 conmon[36358]: cluster 2021-11-20T05:38:32.301150+0000 mgr.smithi058.zagdlk (mgr.14164) 325 : cluster [DBG] pgmap v302: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T05:38:33.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:33 smithi187 conmon[36358]: 2021-11-20T05:38:33.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:33 smithi187 conmon[36358]: cluster 2021-11-20T05:38:32.367759+0000 mon.smithi058 (mon.0) 856 : cluster [DBG] mds.? [v2:172.21.15.187:6826/1745887865,v1:172.21.15.187:6827/1745887865] up:resolve 2021-11-20T05:38:33.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:33 smithi187 conmon[36358]: cluster 2021-11-20T05:38:32.367811+0000 mon.smithi058 (mon.0) 857 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:33.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:33 smithi187 conmon[36358]: audit 2021-11-20T05:38:33.297077+0000 mon.smithi058 (mon.0) 858 : audit [DBG] from='client.? 172.21.15.58:0/296747913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:33.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:33 smithi058 conmon[33537]: cluster 2021-11-20T05:38:32.301150 2021-11-20T05:38:33.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:33 smithi058 conmon[33537]: +0000 mgr.smithi058.zagdlk (mgr.14164) 325 : cluster [DBG] pgmap v302: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T05:38:33.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:33 smithi058 conmon[33537]: cluster 2021-11-20T05:38:32.367759+0000 mon.smithi058 ( 2021-11-20T05:38:33.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:33 smithi058 conmon[33537]: mon.0) 856 : cluster [DBG] mds.? [v2:172.21.15.187:6826/1745887865,v1:172.21.15.187:6827/1745887865] up:resolve 2021-11-20T05:38:33.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:33 smithi058 conmon[33537]: cluster 2021-11-20T05:38: 2021-11-20T05:38:33.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:33 smithi058 conmon[33537]: 32.367811+0000 mon.smithi058 (mon.0) 857 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:33.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:33 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:38:33.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:33 smithi058 conmon[33537]: 11-20T05:38: 2021-11-20T05:38:33.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:33 smithi058 conmon[33537]: 33.297077+0000 mon.smithi058 (mon.0) 2021-11-20T05:38:33.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:33 smithi058 conmon[33537]: 858 : audit [DBG] from='client.? 172.21.15.58:0/296747913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:34.307 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:34.442 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.444+0000 7f7b3ddb9700 1 -- 172.21.15.58:0/2443725556 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7b380f6810 msgr2=0x7f7b380f6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:34.442 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.444+0000 7f7b3ddb9700 1 --2- 172.21.15.58:0/2443725556 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7b380f6810 0x7f7b380f6c70 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f7b2c004660 tx=0x7f7b2c009b30).stop 2021-11-20T05:38:34.443 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.444+0000 7f7b3ddb9700 1 -- 172.21.15.58:0/2443725556 shutdown_connections 2021-11-20T05:38:34.443 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.444+0000 7f7b3ddb9700 1 --2- 172.21.15.58:0/2443725556 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7b380f6810 0x7f7b380f6c70 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:34.443 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.444+0000 7f7b3ddb9700 1 --2- 172.21.15.58:0/2443725556 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b380f5570 0x7f7b380f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:34.444 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.444+0000 7f7b3ddb9700 1 -- 172.21.15.58:0/2443725556 >> 172.21.15.58:0/2443725556 conn(0x7f7b380f0cf0 msgr2=0x7f7b380f3110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:34.444 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.444+0000 7f7b3ddb9700 1 -- 172.21.15.58:0/2443725556 shutdown_connections 2021-11-20T05:38:34.444 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.444+0000 7f7b3ddb9700 1 -- 172.21.15.58:0/2443725556 wait complete. 2021-11-20T05:38:34.444 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.445+0000 7f7b3ddb9700 1 Processor -- start 2021-11-20T05:38:34.445 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.445+0000 7f7b3ddb9700 1 -- start start 2021-11-20T05:38:34.445 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.445+0000 7f7b3ddb9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b380f5570 0x7f7b38105d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:34.445 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.445+0000 7f7b3ddb9700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7b380f6810 0x7f7b38106210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:34.445 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.445+0000 7f7b3ddb9700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b38106810 con 0x7f7b380f6810 2021-11-20T05:38:34.446 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.445+0000 7f7b3ddb9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b38106950 con 0x7f7b380f5570 2021-11-20T05:38:34.446 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.445+0000 7f7b36ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7b380f6810 0x7f7b38106210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:34.446 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.445+0000 7f7b36ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7b380f6810 0x7f7b38106210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42238/0 (socket says 172.21.15.58:42238) 2021-11-20T05:38:34.446 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.445+0000 7f7b36ffd700 1 -- 172.21.15.58:0/3835315978 learned_addr learned my addr 172.21.15.58:0/3835315978 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:34.446 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.445+0000 7f7b377fe700 1 --2- 172.21.15.58:0/3835315978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b380f5570 0x7f7b38105d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:34.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.446+0000 7f7b36ffd700 1 -- 172.21.15.58:0/3835315978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b380f5570 msgr2=0x7f7b38105d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:34.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.446+0000 7f7b36ffd700 1 --2- 172.21.15.58:0/3835315978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b380f5570 0x7f7b38105d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:34.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.446+0000 7f7b36ffd700 1 -- 172.21.15.58:0/3835315978 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c005040 con 0x7f7b380f6810 2021-11-20T05:38:34.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.446+0000 7f7b377fe700 1 --2- 172.21.15.58:0/3835315978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b380f5570 0x7f7b38105d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:38:34.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.446+0000 7f7b36ffd700 1 --2- 172.21.15.58:0/3835315978 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7b380f6810 0x7f7b38106210 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f7b2c000c00 tx=0x7f7b2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:34.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.446+0000 7f7b1ffff700 1 -- 172.21.15.58:0/3835315978 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b2c019070 con 0x7f7b380f6810 2021-11-20T05:38:34.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.446+0000 7f7b1ffff700 1 -- 172.21.15.58:0/3835315978 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b2c007e70 con 0x7f7b380f6810 2021-11-20T05:38:34.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.446+0000 7f7b3ddb9700 1 -- 172.21.15.58:0/3835315978 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3810b300 con 0x7f7b380f6810 2021-11-20T05:38:34.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.446+0000 7f7b1ffff700 1 -- 172.21.15.58:0/3835315978 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b2c014d00 con 0x7f7b380f6810 2021-11-20T05:38:34.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.446+0000 7f7b3ddb9700 1 -- 172.21.15.58:0/3835315978 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3810b860 con 0x7f7b380f6810 2021-11-20T05:38:34.450 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.447+0000 7f7b3ddb9700 1 -- 172.21.15.58:0/3835315978 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b38027a00 con 0x7f7b380f6810 2021-11-20T05:38:34.450 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.447+0000 7f7b1ffff700 1 -- 172.21.15.58:0/3835315978 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7b2c025070 con 0x7f7b380f6810 2021-11-20T05:38:34.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.448+0000 7f7b1ffff700 1 --2- 172.21.15.58:0/3835315978 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7b2005c2a0 0x7f7b2005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:34.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.448+0000 7f7b1ffff700 1 -- 172.21.15.58:0/3835315978 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f7b2c077430 con 0x7f7b380f6810 2021-11-20T05:38:34.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.448+0000 7f7b377fe700 1 --2- 172.21.15.58:0/3835315978 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7b2005c2a0 0x7f7b2005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:34.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.448+0000 7f7b377fe700 1 --2- 172.21.15.58:0/3835315978 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7b2005c2a0 0x7f7b2005e760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f7b38107210 tx=0x7f7b2800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:34.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.452+0000 7f7b1ffff700 1 -- 172.21.15.58:0/3835315978 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b2c049b90 con 0x7f7b380f6810 2021-11-20T05:38:34.645 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.650+0000 7f7b3ddb9700 1 -- 172.21.15.58:0/3835315978 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b380fdd00 con 0x7f7b380f6810 2021-11-20T05:38:34.646 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.651+0000 7f7b1ffff700 1 -- 172.21.15.58:0/3835315978 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2576 (secure 0 0 0) 0x7f7b2c023070 con 0x7f7b380f6810 2021-11-20T05:38:34.647 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:34.647 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:34.369456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24583,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24583":{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:34.648 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.652+0000 7f7b3ddb9700 1 -- 172.21.15.58:0/3835315978 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7b2005c2a0 msgr2=0x7f7b2005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:34.649 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.652+0000 7f7b3ddb9700 1 --2- 172.21.15.58:0/3835315978 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7b2005c2a0 0x7f7b2005e760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f7b38107210 tx=0x7f7b2800b040).stop 2021-11-20T05:38:34.649 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.652+0000 7f7b3ddb9700 1 -- 172.21.15.58:0/3835315978 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7b380f6810 msgr2=0x7f7b38106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:34.649 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.652+0000 7f7b3ddb9700 1 --2- 172.21.15.58:0/3835315978 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7b380f6810 0x7f7b38106210 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f7b2c000c00 tx=0x7f7b2c00b040).stop 2021-11-20T05:38:34.649 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.652+0000 7f7b3ddb9700 1 -- 172.21.15.58:0/3835315978 shutdown_connections 2021-11-20T05:38:34.650 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.652+0000 7f7b3ddb9700 1 --2- 172.21.15.58:0/3835315978 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f7b2005c2a0 0x7f7b2005e760 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:34.650 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.652+0000 7f7b3ddb9700 1 --2- 172.21.15.58:0/3835315978 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b380f5570 0x7f7b38105d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:34.650 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.652+0000 7f7b3ddb9700 1 --2- 172.21.15.58:0/3835315978 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f7b380f6810 0x7f7b38106210 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:34.650 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.652+0000 7f7b3ddb9700 1 -- 172.21.15.58:0/3835315978 >> 172.21.15.58:0/3835315978 conn(0x7f7b380f0cf0 msgr2=0x7f7b380fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:34.651 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.652+0000 7f7b3ddb9700 1 -- 172.21.15.58:0/3835315978 shutdown_connections 2021-11-20T05:38:34.651 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:34.652+0000 7f7b3ddb9700 1 -- 172.21.15.58:0/3835315978 wait complete. 2021-11-20T05:38:34.651 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 95 2021-11-20T05:38:34.659 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:34 smithi058 conmon[33537]: cluster 2021-11-20T05:38:33.368377+0000 mon.smithi058 (mon.0) 859 : cluster [DBG] mds.? [v2:172.21.15.187:6826/1745887865,v1:172.21.15.187:6827/1745887865] up:reconnect 2021-11-20T05:38:34.659 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:34 smithi058 conmon[33537]: cluster 2021-11-20T05:38:33.368443+0000 mon.smithi058 (mon.0) 860 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:34.659 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:34 smithi058 conmon[33537]: cluster 2021-11-20T05:38:33.388167+0000 mds.cephfs.smithi187.fikisa (mds.0) 1 : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0209997 2021-11-20T05:38:34.660 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:34.369456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24583, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 92, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6827/1745887865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 1745887865}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 1745887865}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:34.661 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T05:38:34.685 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:34 smithi187 conmon[36358]: cluster 2021-11-20T05:38:33.368377+0000 mon.smithi058 (mon.0 2021-11-20T05:38:34.685 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:34 smithi187 conmon[36358]: ) 859 : cluster [DBG] mds.? [v2:172.21.15.187:6826/1745887865,v1:172.21.15.187:6827/1745887865] up:reconnect 2021-11-20T05:38:34.685 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:34 smithi187 conmon[36358]: cluster 2021-11-20T05:38:33.368443+0000 mon.smithi058 (mon.0) 860 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:34.686 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:34 smithi187 conmon[36358]: cluster 2021-11-20T05:38:33.388167+0000 mds.cephfs.smithi187.fikisa (mds.0) 1 2021-11-20T05:38:34.686 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:34 smithi187 conmon[36358]: : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0209997 2021-11-20T05:38:35.465 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:35 smithi058 conmon[33537]: cluster 2021-11-20T05:38:34.301653+0000 mgr.smithi058.zagdlk (mgr.14164) 326 : 2021-11-20T05:38:35.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:35 smithi058 conmon[33537]: cluster [DBG] pgmap v303: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:35.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:35 smithi058 conmon[33537]: cluster 2021-11-20T05:38:34.373158+0000 mon.smithi058 (mon.0) 861 : cluster [DBG] mds.? [v2:172.21.15.187:6826/1745887865,v1:172.21.15.187:6827/1745887865] up:rejoin 2021-11-20T05:38:35.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:35 smithi058 conmon[33537]: cluster 2021-11 2021-11-20T05:38:35.467 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:35 smithi058 conmon[33537]: -20T05:38:34.373233+0000 mon.smithi058 (mon.0) 862 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:35.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:35 smithi058 conmon[33537]: audit 2021-11-20T05: 2021-11-20T05:38:35.468 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:35 smithi058 conmon[33537]: 38:34.651649+0000 mon.smithi058 (mon.0) 863 : audit [DBG] from='client.? 172.21.15.58:0/3835315978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:35.662 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:35.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:35 smithi187 conmon[36358]: cluster 2021-11-20T05:38:34.301653+0000 mgr.smithi058.zagdlk (mgr.14164) 326 : cluster 2021-11-20T05:38:35.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:35 smithi187 conmon[36358]: [DBG] pgmap v303: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:35.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:35 smithi187 conmon[36358]: cluster 2021-11-20T05:38:34.373158+0000 mon.smithi058 (mon.0) 861 : cluster [DBG] mds.? [v2:172.21.15.187:6826/1745887865,v1:172.21.15.187:6827/1745887865] up:rejoin 2021-11-20T05:38:35.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:35 smithi187 conmon[36358]: cluster 2021-11-20T05:38:34.373233+0000 mon.smithi058 (mon.0) 862 : cluster 2021-11-20T05:38:35.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:35 smithi187 conmon[36358]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:35.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:35 smithi187 conmon[36358]: audit 2021-11-20T05:38:34 2021-11-20T05:38:35.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:35 smithi187 conmon[36358]: .651649+0000 mon.smithi058 (mon.0) 863 : audit [DBG] 2021-11-20T05:38:35.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:35 smithi187 conmon[36358]: from='client.? 172.21.15.58:0/3835315978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:35.796 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.798+0000 7fa846a63700 1 -- 172.21.15.58:0/1168535256 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa8400f6820 msgr2=0x7fa8400f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:35.796 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.799+0000 7fa846a63700 1 --2- 172.21.15.58:0/1168535256 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa8400f6820 0x7fa8400f6c80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fa834004660 tx=0x7fa834009b30).stop 2021-11-20T05:38:35.797 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.799+0000 7fa846a63700 1 -- 172.21.15.58:0/1168535256 shutdown_connections 2021-11-20T05:38:35.797 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.799+0000 7fa846a63700 1 --2- 172.21.15.58:0/1168535256 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa8400f6820 0x7fa8400f6c80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:35.797 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.799+0000 7fa846a63700 1 --2- 172.21.15.58:0/1168535256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8400f5580 0x7fa8400f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:35.797 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.799+0000 7fa846a63700 1 -- 172.21.15.58:0/1168535256 >> 172.21.15.58:0/1168535256 conn(0x7fa8400f0ce0 msgr2=0x7fa8400f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:35.798 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.799+0000 7fa846a63700 1 -- 172.21.15.58:0/1168535256 shutdown_connections 2021-11-20T05:38:35.798 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.799+0000 7fa846a63700 1 -- 172.21.15.58:0/1168535256 wait complete. 2021-11-20T05:38:35.798 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.799+0000 7fa846a63700 1 Processor -- start 2021-11-20T05:38:35.798 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.799+0000 7fa846a63700 1 -- start start 2021-11-20T05:38:35.799 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.800+0000 7fa846a63700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8400f5580 0x7fa840060530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:35.799 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.800+0000 7fa846a63700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa8400f6820 0x7fa84010e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:35.799 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.800+0000 7fa846a63700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa840107a10 con 0x7fa8400f6820 2021-11-20T05:38:35.799 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.800+0000 7fa846a63700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84010e520 con 0x7fa8400f5580 2021-11-20T05:38:35.799 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.800+0000 7fa83f7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa8400f6820 0x7fa84010e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:35.800 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.800+0000 7fa83f7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa8400f6820 0x7fa84010e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42258/0 (socket says 172.21.15.58:42258) 2021-11-20T05:38:35.800 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.800+0000 7fa83f7fe700 1 -- 172.21.15.58:0/2779868397 learned_addr learned my addr 172.21.15.58:0/2779868397 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:35.800 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.800+0000 7fa83ffff700 1 --2- 172.21.15.58:0/2779868397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8400f5580 0x7fa840060530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:35.800 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.800+0000 7fa83f7fe700 1 -- 172.21.15.58:0/2779868397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8400f5580 msgr2=0x7fa840060530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:35.801 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.800+0000 7fa83f7fe700 1 --2- 172.21.15.58:0/2779868397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8400f5580 0x7fa840060530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:35.801 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.800+0000 7fa83f7fe700 1 -- 172.21.15.58:0/2779868397 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa834005040 con 0x7fa8400f6820 2021-11-20T05:38:35.801 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.800+0000 7fa83f7fe700 1 --2- 172.21.15.58:0/2779868397 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa8400f6820 0x7fa84010e030 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fa834013530 tx=0x7fa83400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:35.801 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.800+0000 7fa83cff9700 1 -- 172.21.15.58:0/2779868397 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa834019070 con 0x7fa8400f6820 2021-11-20T05:38:35.801 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.800+0000 7fa83cff9700 1 -- 172.21.15.58:0/2779868397 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa834007e70 con 0x7fa8400f6820 2021-11-20T05:38:35.802 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.800+0000 7fa846a63700 1 -- 172.21.15.58:0/2779868397 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84010e660 con 0x7fa8400f6820 2021-11-20T05:38:35.802 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.800+0000 7fa83cff9700 1 -- 172.21.15.58:0/2779868397 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa834014c90 con 0x7fa8400f6820 2021-11-20T05:38:35.802 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.800+0000 7fa846a63700 1 -- 172.21.15.58:0/2779868397 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84010eb90 con 0x7fa8400f6820 2021-11-20T05:38:35.803 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.801+0000 7fa846a63700 1 -- 172.21.15.58:0/2779868397 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa840046330 con 0x7fa8400f6820 2021-11-20T05:38:35.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.802+0000 7fa83cff9700 1 -- 172.21.15.58:0/2779868397 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa834025070 con 0x7fa8400f6820 2021-11-20T05:38:35.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.802+0000 7fa83cff9700 1 --2- 172.21.15.58:0/2779868397 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa82805c2a0 0x7fa82805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:35.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.802+0000 7fa83cff9700 1 -- 172.21.15.58:0/2779868397 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fa8340772e0 con 0x7fa8400f6820 2021-11-20T05:38:35.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.802+0000 7fa83ffff700 1 --2- 172.21.15.58:0/2779868397 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa82805c2a0 0x7fa82805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:35.804 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.803+0000 7fa83ffff700 1 --2- 172.21.15.58:0/2779868397 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa82805c2a0 0x7fa82805e760 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fa8400f6680 tx=0x7fa83000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:35.805 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:35.806+0000 7fa83cff9700 1 -- 172.21.15.58:0/2779868397 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa834049b20 con 0x7fa8400f6820 2021-11-20T05:38:36.000 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:36.005+0000 7fa846a63700 1 -- 172.21.15.58:0/2779868397 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa840027a00 con 0x7fa8400f6820 2021-11-20T05:38:36.008 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:36.012+0000 7fa83cff9700 1 -- 172.21.15.58:0/2779868397 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2575 (secure 0 0 0) 0x7fa83404d140 con 0x7fa8400f6820 2021-11-20T05:38:36.008 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:36.008 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:34.369456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24583,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24583":{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:36.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:36.017+0000 7fa846a63700 1 -- 172.21.15.58:0/2779868397 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa82805c2a0 msgr2=0x7fa82805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:36.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:36.017+0000 7fa846a63700 1 --2- 172.21.15.58:0/2779868397 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa82805c2a0 0x7fa82805e760 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fa8400f6680 tx=0x7fa83000b040).stop 2021-11-20T05:38:36.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:36.017+0000 7fa846a63700 1 -- 172.21.15.58:0/2779868397 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa8400f6820 msgr2=0x7fa84010e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:36.014 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:36.017+0000 7fa846a63700 1 --2- 172.21.15.58:0/2779868397 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa8400f6820 0x7fa84010e030 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fa834013530 tx=0x7fa83400b040).stop 2021-11-20T05:38:36.015 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:36.017+0000 7fa846a63700 1 -- 172.21.15.58:0/2779868397 shutdown_connections 2021-11-20T05:38:36.015 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:36.017+0000 7fa846a63700 1 --2- 172.21.15.58:0/2779868397 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa82805c2a0 0x7fa82805e760 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:36.015 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:36.017+0000 7fa846a63700 1 --2- 172.21.15.58:0/2779868397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8400f5580 0x7fa840060530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:36.016 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:36.017+0000 7fa846a63700 1 --2- 172.21.15.58:0/2779868397 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa8400f6820 0x7fa84010e030 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:36.016 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:36.017+0000 7fa846a63700 1 -- 172.21.15.58:0/2779868397 >> 172.21.15.58:0/2779868397 conn(0x7fa8400f0ce0 msgr2=0x7fa8400fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:36.016 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:36.018+0000 7fa846a63700 1 -- 172.21.15.58:0/2779868397 shutdown_connections 2021-11-20T05:38:36.016 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:36.018+0000 7fa846a63700 1 -- 172.21.15.58:0/2779868397 wait complete. 2021-11-20T05:38:36.016 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 96 2021-11-20T05:38:36.027 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:34.369456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24583, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 92, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6827/1745887865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 1745887865}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 1745887865}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:36.027 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T05:38:36.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:36 smithi187 conmon[36358]: cluster 2021-11-20T05:38:35. 2021-11-20T05:38:36.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:36 smithi187 conmon[36358]: 381850+0000 mon.smithi058 (mon.0) 864 : cluster [DBG] mds.? [v2:172.21.15.187:6824/1414418853,v1:172.21.15.187:6825/1414418853] up:standby 2021-11-20T05:38:36.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:36 smithi187 conmon[36358]: cluster 2021-11-20T05:38:35 2021-11-20T05:38:36.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:36 smithi187 conmon[36358]: .381928+0000 mon.smithi058 (mon.0) 865 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:36.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:36 smithi187 conmon[36358]: audit 2021-11-20T05:38:36.009260+0000 mon.smithi058 (mon.0 2021-11-20T05:38:36.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:36 smithi187 conmon[36358]: ) 866 : audit [DBG] from='client.? 172.21.15.58:0/2779868397' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:36.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:36 smithi058 conmon[33537]: cluster 2021-11-20T05:38:35.381850+0000 mon.smithi058 (mon.0) 864 : cluster 2021-11-20T05:38:36.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:36 smithi058 conmon[33537]: [DBG] mds.? [v2:172.21.15.187:6824/1414418853,v1:172.21.15.187:6825/1414418853] up:standby 2021-11-20T05:38:36.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:36 smithi058 conmon[33537]: cluster 2021-11-20T05:38:35.381928+0000 mon.smithi058 (mon.0) 865 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.fikisa=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 1 up:standby 2021-11-20T05:38:36.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:36 smithi058 conmon[33537]: audit 2021-11-20T05:38:36.009260+0000 mon.smithi058 (mon. 2021-11-20T05:38:36.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:36 smithi058 conmon[33537]: 0) 866 : audit [DBG] from='client.? 172.21.15.58:0/2779868397' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:37.029 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:37.164 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.165+0000 7fbf212ae700 1 -- 172.21.15.58:0/132340305 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbf1c0f6820 msgr2=0x7fbf1c0f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:37.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.165+0000 7fbf212ae700 1 --2- 172.21.15.58:0/132340305 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbf1c0f6820 0x7fbf1c0f6c80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7fbf0c004660 tx=0x7fbf0c009b30).stop 2021-11-20T05:38:37.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.165+0000 7fbf212ae700 1 -- 172.21.15.58:0/132340305 shutdown_connections 2021-11-20T05:38:37.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.165+0000 7fbf212ae700 1 --2- 172.21.15.58:0/132340305 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbf1c0f6820 0x7fbf1c0f6c80 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:37.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.165+0000 7fbf212ae700 1 --2- 172.21.15.58:0/132340305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c0f5580 0x7fbf1c0f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:37.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.165+0000 7fbf212ae700 1 -- 172.21.15.58:0/132340305 >> 172.21.15.58:0/132340305 conn(0x7fbf1c0f0ce0 msgr2=0x7fbf1c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:37.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.165+0000 7fbf212ae700 1 -- 172.21.15.58:0/132340305 shutdown_connections 2021-11-20T05:38:37.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.165+0000 7fbf212ae700 1 -- 172.21.15.58:0/132340305 wait complete. 2021-11-20T05:38:37.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.166+0000 7fbf212ae700 1 Processor -- start 2021-11-20T05:38:37.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.166+0000 7fbf212ae700 1 -- start start 2021-11-20T05:38:37.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.166+0000 7fbf212ae700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbf1c0f5580 0x7fbf1c105dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:37.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.166+0000 7fbf212ae700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c0f6820 0x7fbf1c1062b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:37.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.166+0000 7fbf212ae700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c106860 con 0x7fbf1c0f5580 2021-11-20T05:38:37.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.166+0000 7fbf212ae700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c1069a0 con 0x7fbf1c0f6820 2021-11-20T05:38:37.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.166+0000 7fbf1affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbf1c0f5580 0x7fbf1c105dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:37.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.166+0000 7fbf1affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbf1c0f5580 0x7fbf1c105dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42268/0 (socket says 172.21.15.58:42268) 2021-11-20T05:38:37.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.166+0000 7fbf1affd700 1 -- 172.21.15.58:0/473390133 learned_addr learned my addr 172.21.15.58:0/473390133 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:37.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.166+0000 7fbf1a7fc700 1 --2- 172.21.15.58:0/473390133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c0f6820 0x7fbf1c1062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:37.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.167+0000 7fbf1affd700 1 -- 172.21.15.58:0/473390133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c0f6820 msgr2=0x7fbf1c1062b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:37.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.167+0000 7fbf1affd700 1 --2- 172.21.15.58:0/473390133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c0f6820 0x7fbf1c1062b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:37.170 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.167+0000 7fbf1affd700 1 -- 172.21.15.58:0/473390133 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c005040 con 0x7fbf1c0f5580 2021-11-20T05:38:37.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.167+0000 7fbf1a7fc700 1 --2- 172.21.15.58:0/473390133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c0f6820 0x7fbf1c1062b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:38:37.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.167+0000 7fbf1affd700 1 --2- 172.21.15.58:0/473390133 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbf1c0f5580 0x7fbf1c105dc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fbf04002700 tx=0x7fbf04009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:37.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.167+0000 7fbf137fe700 1 -- 172.21.15.58:0/473390133 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf04009320 con 0x7fbf1c0f5580 2021-11-20T05:38:37.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.167+0000 7fbf137fe700 1 -- 172.21.15.58:0/473390133 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf0400f040 con 0x7fbf1c0f5580 2021-11-20T05:38:37.171 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.167+0000 7fbf212ae700 1 -- 172.21.15.58:0/473390133 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf1c10b440 con 0x7fbf1c0f5580 2021-11-20T05:38:37.172 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.167+0000 7fbf137fe700 1 -- 172.21.15.58:0/473390133 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf04014900 con 0x7fbf1c0f5580 2021-11-20T05:38:37.172 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.167+0000 7fbf212ae700 1 -- 172.21.15.58:0/473390133 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf1c10ba00 con 0x7fbf1c0f5580 2021-11-20T05:38:37.172 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.168+0000 7fbf212ae700 1 -- 172.21.15.58:0/473390133 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf1c027a00 con 0x7fbf1c0f5580 2021-11-20T05:38:37.172 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.168+0000 7fbf137fe700 1 -- 172.21.15.58:0/473390133 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbf04009480 con 0x7fbf1c0f5580 2021-11-20T05:38:37.173 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.169+0000 7fbf137fe700 1 --2- 172.21.15.58:0/473390133 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbf0805c2a0 0x7fbf0805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:37.173 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.169+0000 7fbf137fe700 1 -- 172.21.15.58:0/473390133 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fbf04076720 con 0x7fbf1c0f5580 2021-11-20T05:38:37.173 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.169+0000 7fbf1a7fc700 1 --2- 172.21.15.58:0/473390133 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbf0805c2a0 0x7fbf0805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:37.174 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.169+0000 7fbf1a7fc700 1 --2- 172.21.15.58:0/473390133 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbf0805c2a0 0x7fbf0805e760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fbf0c005010 tx=0x7fbf0c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:37.174 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.173+0000 7fbf137fe700 1 -- 172.21.15.58:0/473390133 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf04048ee0 con 0x7fbf1c0f5580 2021-11-20T05:38:37.385 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.389+0000 7fbf212ae700 1 -- 172.21.15.58:0/473390133 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf1c0586e0 con 0x7fbf1c0f5580 2021-11-20T05:38:37.385 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.390+0000 7fbf137fe700 1 -- 172.21.15.58:0/473390133 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2575 (secure 0 0 0) 0x7fbf0404c500 con 0x7fbf1c0f5580 2021-11-20T05:38:37.386 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:37.386 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:34.369456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24583,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24583":{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:37.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.391+0000 7fbf212ae700 1 -- 172.21.15.58:0/473390133 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbf0805c2a0 msgr2=0x7fbf0805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:37.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.391+0000 7fbf212ae700 1 --2- 172.21.15.58:0/473390133 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbf0805c2a0 0x7fbf0805e760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fbf0c005010 tx=0x7fbf0c009b30).stop 2021-11-20T05:38:37.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.391+0000 7fbf212ae700 1 -- 172.21.15.58:0/473390133 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbf1c0f5580 msgr2=0x7fbf1c105dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:37.388 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.391+0000 7fbf212ae700 1 --2- 172.21.15.58:0/473390133 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbf1c0f5580 0x7fbf1c105dc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fbf04002700 tx=0x7fbf04009d80).stop 2021-11-20T05:38:37.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.392+0000 7fbf212ae700 1 -- 172.21.15.58:0/473390133 shutdown_connections 2021-11-20T05:38:37.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.392+0000 7fbf212ae700 1 --2- 172.21.15.58:0/473390133 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fbf0805c2a0 0x7fbf0805e760 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:37.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.392+0000 7fbf212ae700 1 --2- 172.21.15.58:0/473390133 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fbf1c0f5580 0x7fbf1c105dc0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:37.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.392+0000 7fbf212ae700 1 --2- 172.21.15.58:0/473390133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf1c0f6820 0x7fbf1c1062b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:37.389 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.392+0000 7fbf212ae700 1 -- 172.21.15.58:0/473390133 >> 172.21.15.58:0/473390133 conn(0x7fbf1c0f0ce0 msgr2=0x7fbf1c0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:37.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.392+0000 7fbf212ae700 1 -- 172.21.15.58:0/473390133 shutdown_connections 2021-11-20T05:38:37.390 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:37.392+0000 7fbf212ae700 1 -- 172.21.15.58:0/473390133 wait complete. 2021-11-20T05:38:37.390 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 96 2021-11-20T05:38:37.399 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:34.369456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24583, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 92, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6827/1745887865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 1745887865}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 1745887865}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:37.400 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T05:38:37.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:37 smithi187 conmon[36358]: cluster 2021-11-20T05:38:36.302096+0000 mgr.smithi058.zagdlk 2021-11-20T05:38:37.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:37 smithi187 conmon[36358]: (mgr.14164) 327 : cluster [DBG] pgmap v304: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:37.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:37 smithi058 conmon[33537]: cluster 2021-11-20T05:38:36.302096+0000 mgr.smithi058.zagdlk 2021-11-20T05:38:37.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:37 smithi058 conmon[33537]: (mgr.14164) 327 : cluster [DBG] pgmap v304: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:38.401 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:38.537 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.539+0000 7f4ac3b30700 1 -- 172.21.15.58:0/1733811707 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4abc0f6670 msgr2=0x7f4abc0f6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:38.538 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.539+0000 7f4ac3b30700 1 --2- 172.21.15.58:0/1733811707 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4abc0f6670 0x7f4abc0f6a90 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f4ab8004660 tx=0x7f4ab8009b30).stop 2021-11-20T05:38:38.538 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.539+0000 7f4ac3b30700 1 -- 172.21.15.58:0/1733811707 shutdown_connections 2021-11-20T05:38:38.538 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.539+0000 7f4ac3b30700 1 --2- 172.21.15.58:0/1733811707 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4abc0f7910 0x7f4abc0f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:38.538 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.539+0000 7f4ac3b30700 1 --2- 172.21.15.58:0/1733811707 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4abc0f6670 0x7f4abc0f6a90 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:38.538 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.539+0000 7f4ac3b30700 1 -- 172.21.15.58:0/1733811707 >> 172.21.15.58:0/1733811707 conn(0x7f4abc0f1db0 msgr2=0x7f4abc0f41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:38.539 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.539+0000 7f4ac3b30700 1 -- 172.21.15.58:0/1733811707 shutdown_connections 2021-11-20T05:38:38.539 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.539+0000 7f4ac3b30700 1 -- 172.21.15.58:0/1733811707 wait complete. 2021-11-20T05:38:38.539 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.540+0000 7f4ac3b30700 1 Processor -- start 2021-11-20T05:38:38.539 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.540+0000 7f4ac3b30700 1 -- start start 2021-11-20T05:38:38.540 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.540+0000 7f4ac3b30700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4abc0f6670 0x7f4abc106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:38.540 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.540+0000 7f4ac3b30700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4abc0f7910 0x7f4abc1073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:38.540 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.540+0000 7f4ac3b30700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc107960 con 0x7f4abc0f6670 2021-11-20T05:38:38.540 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.540+0000 7f4ac3b30700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc107aa0 con 0x7f4abc0f7910 2021-11-20T05:38:38.541 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.540+0000 7f4ac18cc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4abc0f6670 0x7f4abc106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:38.541 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.540+0000 7f4ac18cc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4abc0f6670 0x7f4abc106ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42282/0 (socket says 172.21.15.58:42282) 2021-11-20T05:38:38.541 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.540+0000 7f4ac18cc700 1 -- 172.21.15.58:0/1439643748 learned_addr learned my addr 172.21.15.58:0/1439643748 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:38.541 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.540+0000 7f4ac10cb700 1 --2- 172.21.15.58:0/1439643748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4abc0f7910 0x7f4abc1073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:38.541 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.540+0000 7f4ac18cc700 1 -- 172.21.15.58:0/1439643748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4abc0f7910 msgr2=0x7f4abc1073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:38.542 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.540+0000 7f4ac18cc700 1 --2- 172.21.15.58:0/1439643748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4abc0f7910 0x7f4abc1073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:38.542 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.540+0000 7f4ac18cc700 1 -- 172.21.15.58:0/1439643748 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab8005040 con 0x7f4abc0f6670 2021-11-20T05:38:38.542 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.541+0000 7f4ac18cc700 1 --2- 172.21.15.58:0/1439643748 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4abc0f6670 0x7f4abc106ec0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f4ab8007a70 tx=0x7f4ab8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:38.542 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.541+0000 7f4ab27fc700 1 -- 172.21.15.58:0/1439643748 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ab8018070 con 0x7f4abc0f6670 2021-11-20T05:38:38.543 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.541+0000 7f4ab27fc700 1 -- 172.21.15.58:0/1439643748 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ab8009700 con 0x7f4abc0f6670 2021-11-20T05:38:38.543 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.541+0000 7f4ac3b30700 1 -- 172.21.15.58:0/1439643748 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4abc10c4e0 con 0x7f4abc0f6670 2021-11-20T05:38:38.543 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.541+0000 7f4ab27fc700 1 -- 172.21.15.58:0/1439643748 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ab801c620 con 0x7f4abc0f6670 2021-11-20T05:38:38.543 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.541+0000 7f4ac3b30700 1 -- 172.21.15.58:0/1439643748 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4abc10caa0 con 0x7f4abc0f6670 2021-11-20T05:38:38.543 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.542+0000 7f4ac3b30700 1 -- 172.21.15.58:0/1439643748 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4abc027a00 con 0x7f4abc0f6670 2021-11-20T05:38:38.544 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.542+0000 7f4ab27fc700 1 -- 172.21.15.58:0/1439643748 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4ab801d420 con 0x7f4abc0f6670 2021-11-20T05:38:38.545 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.543+0000 7f4ab27fc700 1 --2- 172.21.15.58:0/1439643748 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4aa805c2a0 0x7f4aa805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:38.545 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.543+0000 7f4ab27fc700 1 -- 172.21.15.58:0/1439643748 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f4ab80782b0 con 0x7f4abc0f6670 2021-11-20T05:38:38.546 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.543+0000 7f4ac10cb700 1 --2- 172.21.15.58:0/1439643748 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4aa805c2a0 0x7f4aa805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:38.546 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.543+0000 7f4ac10cb700 1 --2- 172.21.15.58:0/1439643748 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4aa805c2a0 0x7f4aa805e760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f4aac007fd0 tx=0x7f4aac009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:38.546 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.547+0000 7f4ab27fc700 1 -- 172.21.15.58:0/1439643748 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ab804aa70 con 0x7f4abc0f6670 2021-11-20T05:38:38.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:38 smithi187 conmon[36358]: audit 2021-11-20T05:38:37.390928+0000 mon.smithi058 (mon.0) 867 : audit 2021-11-20T05:38:38.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:38 smithi187 conmon[36358]: [DBG] from='client.? 172.21.15.58:0/473390133' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:38.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:38 smithi058 conmon[33537]: audit 2021-11-20T05:38:37.390928+0000 mon.smithi058 (mon.0) 867 : audit 2021-11-20T05:38:38.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:38 smithi058 conmon[33537]: [DBG] from='client.? 172.21.15.58:0/473390133' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:38.739 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.744+0000 7f4ac3b30700 1 -- 172.21.15.58:0/1439643748 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4abc10c670 con 0x7f4abc0f6670 2021-11-20T05:38:38.740 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.744+0000 7f4ab27fc700 1 -- 172.21.15.58:0/1439643748 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f4abc10c670 con 0x7f4abc0f6670 2021-11-20T05:38:38.740 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:38.740 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:34.369456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24583,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24583":{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:38.742 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.746+0000 7f4ac3b30700 1 -- 172.21.15.58:0/1439643748 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4aa805c2a0 msgr2=0x7f4aa805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:38.742 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.746+0000 7f4ac3b30700 1 --2- 172.21.15.58:0/1439643748 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4aa805c2a0 0x7f4aa805e760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f4aac007fd0 tx=0x7f4aac009040).stop 2021-11-20T05:38:38.742 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.746+0000 7f4ac3b30700 1 -- 172.21.15.58:0/1439643748 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4abc0f6670 msgr2=0x7f4abc106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:38.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.746+0000 7f4ac3b30700 1 --2- 172.21.15.58:0/1439643748 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4abc0f6670 0x7f4abc106ec0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f4ab8007a70 tx=0x7f4ab8009b30).stop 2021-11-20T05:38:38.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.746+0000 7f4ac3b30700 1 -- 172.21.15.58:0/1439643748 shutdown_connections 2021-11-20T05:38:38.743 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.746+0000 7f4ac3b30700 1 --2- 172.21.15.58:0/1439643748 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f4aa805c2a0 0x7f4aa805e760 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:38.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.746+0000 7f4ac3b30700 1 --2- 172.21.15.58:0/1439643748 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f4abc0f6670 0x7f4abc106ec0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:38.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.746+0000 7f4ac3b30700 1 --2- 172.21.15.58:0/1439643748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4abc0f7910 0x7f4abc1073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:38.744 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.746+0000 7f4ac3b30700 1 -- 172.21.15.58:0/1439643748 >> 172.21.15.58:0/1439643748 conn(0x7f4abc0f1db0 msgr2=0x7f4abc0fb360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:38.745 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.746+0000 7f4ac3b30700 1 -- 172.21.15.58:0/1439643748 shutdown_connections 2021-11-20T05:38:38.745 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:38.746+0000 7f4ac3b30700 1 -- 172.21.15.58:0/1439643748 wait complete. 2021-11-20T05:38:38.745 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 96 2021-11-20T05:38:38.753 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:34.369456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24583, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 92, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6827/1745887865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 1745887865}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 1745887865}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:38.754 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T05:38:39.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:39 smithi187 conmon[36358]: cluster 2021-11-20T05:38:38.302785+0000 mgr.smithi058.zagdlk (mgr.14164) 328 : 2021-11-20T05:38:39.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:39 smithi187 conmon[36358]: cluster [DBG] pgmap v305: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:39.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:39 smithi187 conmon[36358]: audit 2021-11-20T05:38:38.745251+0000 mon.smithi058 (mon.0) 868 : audit [DBG] from='client.? 172.21.15.58:0/1439643748' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:39.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:39 smithi058 conmon[33537]: cluster 2021-11-20T05:38:38.302785+0000 mgr.smithi058.zagdlk (mgr.14164) 328 2021-11-20T05:38:39.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:39 smithi058 conmon[33537]: : cluster [DBG] pgmap v305: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:39.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:39 smithi058 conmon[33537]: audit 2021-11-20T05:38:38.745251+0000 mon.smithi058 (mon.0) 868 : audit [DBG] from='client.? 172.21.15.58:0/1439643748' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:39.755 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:39.890 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.893+0000 7f72348cf700 1 -- 172.21.15.58:0/2102895813 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72300f4d40 msgr2=0x7f72300f5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:39.891 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.893+0000 7f72348cf700 1 --2- 172.21.15.58:0/2102895813 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72300f4d40 0x7f72300f5160 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f7220004660 tx=0x7f7220009b30).stop 2021-11-20T05:38:39.891 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.893+0000 7f72348cf700 1 -- 172.21.15.58:0/2102895813 shutdown_connections 2021-11-20T05:38:39.891 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.893+0000 7f72348cf700 1 --2- 172.21.15.58:0/2102895813 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72300f5fe0 0x7f72300f6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:39.891 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.893+0000 7f72348cf700 1 --2- 172.21.15.58:0/2102895813 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72300f4d40 0x7f72300f5160 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:39.892 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.893+0000 7f72348cf700 1 -- 172.21.15.58:0/2102895813 >> 172.21.15.58:0/2102895813 conn(0x7f72300f04a0 msgr2=0x7f72300f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:39.892 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.893+0000 7f72348cf700 1 -- 172.21.15.58:0/2102895813 shutdown_connections 2021-11-20T05:38:39.892 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.893+0000 7f72348cf700 1 -- 172.21.15.58:0/2102895813 wait complete. 2021-11-20T05:38:39.892 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.893+0000 7f72348cf700 1 Processor -- start 2021-11-20T05:38:39.892 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.893+0000 7f72348cf700 1 -- start start 2021-11-20T05:38:39.893 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.893+0000 7f72348cf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72300f4d40 0x7f72300f9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:39.893 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.893+0000 7f72348cf700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72300f5fe0 0x7f72300f8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:39.893 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.893+0000 7f72348cf700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72300fa380 con 0x7f72300f5fe0 2021-11-20T05:38:39.893 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.893+0000 7f72348cf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72300f8920 con 0x7f72300f4d40 2021-11-20T05:38:39.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.894+0000 7f722dd9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72300f5fe0 0x7f72300f8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:39.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.894+0000 7f722dd9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72300f5fe0 0x7f72300f8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42296/0 (socket says 172.21.15.58:42296) 2021-11-20T05:38:39.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.894+0000 7f722dd9b700 1 -- 172.21.15.58:0/856601788 learned_addr learned my addr 172.21.15.58:0/856601788 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:39.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.894+0000 7f722dd9b700 1 -- 172.21.15.58:0/856601788 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72300f4d40 msgr2=0x7f72300f9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:39.899 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.894+0000 7f722dd9b700 1 --2- 172.21.15.58:0/856601788 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72300f4d40 0x7f72300f9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:38:39.899 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.894+0000 7f722dd9b700 1 -- 172.21.15.58:0/856601788 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7220005040 con 0x7f72300f5fe0 2021-11-20T05:38:39.899 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.894+0000 7f722dd9b700 1 --2- 172.21.15.58:0/856601788 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72300f5fe0 0x7f72300f8430 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f722400c530 tx=0x7f7224007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:39.900 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.894+0000 7f721effd700 1 -- 172.21.15.58:0/856601788 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f722400f650 con 0x7f72300f5fe0 2021-11-20T05:38:39.900 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.894+0000 7f721effd700 1 -- 172.21.15.58:0/856601788 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f722400fc60 con 0x7f72300f5fe0 2021-11-20T05:38:39.900 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.894+0000 7f721effd700 1 -- 172.21.15.58:0/856601788 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7224012c80 con 0x7f72300f5fe0 2021-11-20T05:38:39.901 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.894+0000 7f72348cf700 1 -- 172.21.15.58:0/856601788 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72300f8ba0 con 0x7f72300f5fe0 2021-11-20T05:38:39.901 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.894+0000 7f72348cf700 1 -- 172.21.15.58:0/856601788 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72300f9160 con 0x7f72300f5fe0 2021-11-20T05:38:39.902 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.895+0000 7f72348cf700 1 -- 172.21.15.58:0/856601788 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7230027a00 con 0x7f72300f5fe0 2021-11-20T05:38:39.903 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.896+0000 7f721effd700 1 -- 172.21.15.58:0/856601788 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7224010070 con 0x7f72300f5fe0 2021-11-20T05:38:39.904 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.896+0000 7f721effd700 1 --2- 172.21.15.58:0/856601788 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f721805c2a0 0x7f721805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:39.904 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.896+0000 7f721effd700 1 -- 172.21.15.58:0/856601788 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f7224076040 con 0x7f72300f5fe0 2021-11-20T05:38:39.904 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.896+0000 7f722e59c700 1 --2- 172.21.15.58:0/856601788 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f721805c2a0 0x7f721805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:39.904 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.897+0000 7f722e59c700 1 --2- 172.21.15.58:0/856601788 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f721805c2a0 0x7f721805e760 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f7220009f50 tx=0x7f7220009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:39.904 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:39.900+0000 7f721effd700 1 -- 172.21.15.58:0/856601788 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72240452b0 con 0x7f72300f5fe0 2021-11-20T05:38:40.091 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:40.095+0000 7f72348cf700 1 -- 172.21.15.58:0/856601788 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72300fd4d0 con 0x7f72300f5fe0 2021-11-20T05:38:40.091 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:40.096+0000 7f721effd700 1 -- 172.21.15.58:0/856601788 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f722404c2b0 con 0x7f72300f5fe0 2021-11-20T05:38:40.092 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:40.092 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:34.369456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24583,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24583":{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:40.094 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:40.097+0000 7f72348cf700 1 -- 172.21.15.58:0/856601788 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f721805c2a0 msgr2=0x7f721805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:40.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:40.097+0000 7f72348cf700 1 --2- 172.21.15.58:0/856601788 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f721805c2a0 0x7f721805e760 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f7220009f50 tx=0x7f7220009b30).stop 2021-11-20T05:38:40.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:40.097+0000 7f72348cf700 1 -- 172.21.15.58:0/856601788 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72300f5fe0 msgr2=0x7f72300f8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:40.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:40.097+0000 7f72348cf700 1 --2- 172.21.15.58:0/856601788 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72300f5fe0 0x7f72300f8430 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f722400c530 tx=0x7f7224007e20).stop 2021-11-20T05:38:40.095 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:40.097+0000 7f72348cf700 1 -- 172.21.15.58:0/856601788 shutdown_connections 2021-11-20T05:38:40.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:40.097+0000 7f72348cf700 1 --2- 172.21.15.58:0/856601788 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f721805c2a0 0x7f721805e760 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:40.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:40.097+0000 7f72348cf700 1 --2- 172.21.15.58:0/856601788 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72300f4d40 0x7f72300f9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:38:40.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:40.097+0000 7f72348cf700 1 --2- 172.21.15.58:0/856601788 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f72300f5fe0 0x7f72300f8430 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:40.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:40.097+0000 7f72348cf700 1 -- 172.21.15.58:0/856601788 >> 172.21.15.58:0/856601788 conn(0x7f72300f04a0 msgr2=0x7f72300fa520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:40.096 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:40.097+0000 7f72348cf700 1 -- 172.21.15.58:0/856601788 shutdown_connections 2021-11-20T05:38:40.097 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:40.097+0000 7f72348cf700 1 -- 172.21.15.58:0/856601788 wait complete. 2021-11-20T05:38:40.097 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 96 2021-11-20T05:38:40.105 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:34.369456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24583, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 92, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6827/1745887865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 1745887865}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 1745887865}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:40.105 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T05:38:40.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:40 smithi187 conmon[36358]: audit 2021-11-20T05:38:40.096919+0000 mon.smithi058 (mon.0) 869 2021-11-20T05:38:40.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:40 smithi187 conmon[36358]: : audit [DBG] from='client.? 172.21.15.58:0/856601788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:40.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:40 smithi058 conmon[33537]: audit 2021-11-20T05:38:40.096919+0000 mon.smithi058 (mon. 2021-11-20T05:38:40.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:40 smithi058 conmon[33537]: 0) 869 : audit [DBG] from='client.? 172.21.15.58:0/856601788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:41.106 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:41.241 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.243+0000 7f49378a8700 1 -- 172.21.15.58:0/3907931597 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f49300f5580 msgr2=0x7f49300f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:41.241 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.243+0000 7f49378a8700 1 --2- 172.21.15.58:0/3907931597 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f49300f5580 0x7f49300f59a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f492c004660 tx=0x7f492c009b30).stop 2021-11-20T05:38:41.242 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.243+0000 7f49378a8700 1 -- 172.21.15.58:0/3907931597 shutdown_connections 2021-11-20T05:38:41.242 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.243+0000 7f49378a8700 1 --2- 172.21.15.58:0/3907931597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49300f6820 0x7f49300f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:38:41.243 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.243+0000 7f49378a8700 1 --2- 172.21.15.58:0/3907931597 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f49300f5580 0x7f49300f59a0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:41.243 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.243+0000 7f49378a8700 1 -- 172.21.15.58:0/3907931597 >> 172.21.15.58:0/3907931597 conn(0x7f49300f0ce0 msgr2=0x7f49300f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:41.243 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.243+0000 7f49378a8700 1 -- 172.21.15.58:0/3907931597 shutdown_connections 2021-11-20T05:38:41.243 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.243+0000 7f49378a8700 1 -- 172.21.15.58:0/3907931597 wait complete. 2021-11-20T05:38:41.244 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.244+0000 7f49378a8700 1 Processor -- start 2021-11-20T05:38:41.244 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.244+0000 7f49378a8700 1 -- start start 2021-11-20T05:38:41.244 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.244+0000 7f49378a8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49300f5580 0x7f49300604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:41.244 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.244+0000 7f49378a8700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f49300f6820 0x7f493010bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:41.244 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.244+0000 7f49378a8700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493010c180 con 0x7f49300f6820 2021-11-20T05:38:41.245 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.244+0000 7f49378a8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493010c2c0 con 0x7f49300f5580 2021-11-20T05:38:41.245 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.244+0000 7f4934e43700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f49300f6820 0x7f493010bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:41.245 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.244+0000 7f4934e43700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f49300f6820 0x7f493010bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42308/0 (socket says 172.21.15.58:42308) 2021-11-20T05:38:41.245 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.244+0000 7f4934e43700 1 -- 172.21.15.58:0/810998476 learned_addr learned my addr 172.21.15.58:0/810998476 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:41.246 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.244+0000 7f4935644700 1 --2- 172.21.15.58:0/810998476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49300f5580 0x7f49300604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:41.246 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.244+0000 7f4934e43700 1 -- 172.21.15.58:0/810998476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49300f5580 msgr2=0x7f49300604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:41.246 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.244+0000 7f4934e43700 1 --2- 172.21.15.58:0/810998476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49300f5580 0x7f49300604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:41.246 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.244+0000 7f4934e43700 1 -- 172.21.15.58:0/810998476 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c005040 con 0x7f49300f6820 2021-11-20T05:38:41.247 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.244+0000 7f4935644700 1 --2- 172.21.15.58:0/810998476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49300f5580 0x7f49300604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T05:38:41.247 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.245+0000 7f4934e43700 1 --2- 172.21.15.58:0/810998476 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f49300f6820 0x7f493010bc90 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f492000b410 tx=0x7f4920006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:41.247 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.245+0000 7f4925ffb700 1 -- 172.21.15.58:0/810998476 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49200125f0 con 0x7f49300f6820 2021-11-20T05:38:41.247 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.245+0000 7f4925ffb700 1 -- 172.21.15.58:0/810998476 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4920012c00 con 0x7f49300f6820 2021-11-20T05:38:41.247 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.245+0000 7f49378a8700 1 -- 172.21.15.58:0/810998476 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f493010c5a0 con 0x7f49300f6820 2021-11-20T05:38:41.248 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.245+0000 7f4925ffb700 1 -- 172.21.15.58:0/810998476 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4920002aa0 con 0x7f49300f6820 2021-11-20T05:38:41.248 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.245+0000 7f49378a8700 1 -- 172.21.15.58:0/810998476 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f493010cb60 con 0x7f49300f6820 2021-11-20T05:38:41.248 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.246+0000 7f49378a8700 1 -- 172.21.15.58:0/810998476 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4930027a00 con 0x7f49300f6820 2021-11-20T05:38:41.249 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.246+0000 7f4925ffb700 1 -- 172.21.15.58:0/810998476 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f492000f030 con 0x7f49300f6820 2021-11-20T05:38:41.250 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.247+0000 7f4925ffb700 1 --2- 172.21.15.58:0/810998476 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f491c05c2a0 0x7f491c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:41.250 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.247+0000 7f4925ffb700 1 -- 172.21.15.58:0/810998476 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f49200768a0 con 0x7f49300f6820 2021-11-20T05:38:41.250 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.247+0000 7f4935644700 1 --2- 172.21.15.58:0/810998476 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f491c05c2a0 0x7f491c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:41.250 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.247+0000 7f4935644700 1 --2- 172.21.15.58:0/810998476 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f491c05c2a0 0x7f491c05e760 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f492c009d60 tx=0x7f492c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:41.251 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.251+0000 7f4925ffb700 1 -- 172.21.15.58:0/810998476 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4920049000 con 0x7f49300f6820 2021-11-20T05:38:41.444 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.449+0000 7f49378a8700 1 -- 172.21.15.58:0/810998476 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49300fdd90 con 0x7f49300f6820 2021-11-20T05:38:41.445 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.449+0000 7f4925ffb700 1 -- 172.21.15.58:0/810998476 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f492004c620 con 0x7f49300f6820 2021-11-20T05:38:41.445 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:41.445 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:34.369456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24583,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24583":{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:41.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.451+0000 7f49378a8700 1 -- 172.21.15.58:0/810998476 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f491c05c2a0 msgr2=0x7f491c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:41.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.451+0000 7f49378a8700 1 --2- 172.21.15.58:0/810998476 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f491c05c2a0 0x7f491c05e760 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f492c009d60 tx=0x7f492c009b30).stop 2021-11-20T05:38:41.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.451+0000 7f49378a8700 1 -- 172.21.15.58:0/810998476 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f49300f6820 msgr2=0x7f493010bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:41.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.451+0000 7f49378a8700 1 --2- 172.21.15.58:0/810998476 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f49300f6820 0x7f493010bc90 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f492000b410 tx=0x7f4920006ef0).stop 2021-11-20T05:38:41.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.451+0000 7f49378a8700 1 -- 172.21.15.58:0/810998476 shutdown_connections 2021-11-20T05:38:41.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.451+0000 7f49378a8700 1 --2- 172.21.15.58:0/810998476 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f491c05c2a0 0x7f491c05e760 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:41.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.451+0000 7f49378a8700 1 --2- 172.21.15.58:0/810998476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49300f5580 0x7f49300604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:41.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.451+0000 7f49378a8700 1 --2- 172.21.15.58:0/810998476 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f49300f6820 0x7f493010bc90 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:41.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.451+0000 7f49378a8700 1 -- 172.21.15.58:0/810998476 >> 172.21.15.58:0/810998476 conn(0x7f49300f0ce0 msgr2=0x7f49300fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:41.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.451+0000 7f49378a8700 1 -- 172.21.15.58:0/810998476 shutdown_connections 2021-11-20T05:38:41.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:41.451+0000 7f49378a8700 1 -- 172.21.15.58:0/810998476 wait complete. 2021-11-20T05:38:41.450 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 96 2021-11-20T05:38:41.458 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:34.369456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24583, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 92, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6827/1745887865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 1745887865}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 1745887865}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:41.459 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T05:38:41.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:41 smithi187 conmon[36358]: cluster 2021-11-20T05:38:40.303293+0000 mgr.smithi058.zagdlk (mgr 2021-11-20T05:38:41.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:41 smithi187 conmon[36358]: .14164) 329 : cluster [DBG] pgmap v306: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:41.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:41 smithi058 conmon[33537]: cluster 2021-11-20T05:38:40.303293+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:38:41.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:41 smithi058 conmon[33537]: ) 329 : cluster [DBG] pgmap v306: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:42.460 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:42.596 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.598+0000 7f8b74b1c700 1 -- 172.21.15.58:0/585263590 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b700f5580 msgr2=0x7f8b700f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:42.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.598+0000 7f8b74b1c700 1 --2- 172.21.15.58:0/585263590 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b700f5580 0x7f8b700f59a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f8b60004660 tx=0x7f8b60009b30).stop 2021-11-20T05:38:42.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.598+0000 7f8b74b1c700 1 -- 172.21.15.58:0/585263590 shutdown_connections 2021-11-20T05:38:42.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.598+0000 7f8b74b1c700 1 --2- 172.21.15.58:0/585263590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b700f6820 0x7f8b700f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:42.597 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.598+0000 7f8b74b1c700 1 --2- 172.21.15.58:0/585263590 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b700f5580 0x7f8b700f59a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:42.598 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.598+0000 7f8b74b1c700 1 -- 172.21.15.58:0/585263590 >> 172.21.15.58:0/585263590 conn(0x7f8b700f0ce0 msgr2=0x7f8b700f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:42.598 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.598+0000 7f8b74b1c700 1 -- 172.21.15.58:0/585263590 shutdown_connections 2021-11-20T05:38:42.598 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.598+0000 7f8b74b1c700 1 -- 172.21.15.58:0/585263590 wait complete. 2021-11-20T05:38:42.598 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.598+0000 7f8b74b1c700 1 Processor -- start 2021-11-20T05:38:42.599 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.598+0000 7f8b74b1c700 1 -- start start 2021-11-20T05:38:42.599 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.598+0000 7f8b74b1c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b700f5580 0x7f8b70106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:42.599 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.598+0000 7f8b74b1c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b700f6820 0x7f8b701073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:42.599 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.598+0000 7f8b74b1c700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b70107950 con 0x7f8b700f6820 2021-11-20T05:38:42.600 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.598+0000 7f8b74b1c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b70107a90 con 0x7f8b700f5580 2021-11-20T05:38:42.600 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.599+0000 7f8b6dd9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b700f6820 0x7f8b701073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:42.600 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.599+0000 7f8b6dd9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b700f6820 0x7f8b701073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42318/0 (socket says 172.21.15.58:42318) 2021-11-20T05:38:42.600 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.599+0000 7f8b6dd9b700 1 -- 172.21.15.58:0/4203355429 learned_addr learned my addr 172.21.15.58:0/4203355429 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:42.600 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.599+0000 7f8b6e59c700 1 --2- 172.21.15.58:0/4203355429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b700f5580 0x7f8b70106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:42.601 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.599+0000 7f8b6dd9b700 1 -- 172.21.15.58:0/4203355429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b700f5580 msgr2=0x7f8b70106eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:42.601 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.599+0000 7f8b6dd9b700 1 --2- 172.21.15.58:0/4203355429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b700f5580 0x7f8b70106eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:42.601 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.599+0000 7f8b6dd9b700 1 -- 172.21.15.58:0/4203355429 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b60005040 con 0x7f8b700f6820 2021-11-20T05:38:42.601 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.599+0000 7f8b6dd9b700 1 --2- 172.21.15.58:0/4203355429 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b700f6820 0x7f8b701073a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f8b640026d0 tx=0x7f8b64009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:42.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.599+0000 7f8b5effd700 1 -- 172.21.15.58:0/4203355429 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b64009320 con 0x7f8b700f6820 2021-11-20T05:38:42.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.599+0000 7f8b5effd700 1 -- 172.21.15.58:0/4203355429 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b6400f040 con 0x7f8b700f6820 2021-11-20T05:38:42.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.599+0000 7f8b74b1c700 1 -- 172.21.15.58:0/4203355429 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b7010c530 con 0x7f8b700f6820 2021-11-20T05:38:42.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.599+0000 7f8b5effd700 1 -- 172.21.15.58:0/4203355429 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b640147e0 con 0x7f8b700f6820 2021-11-20T05:38:42.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.599+0000 7f8b74b1c700 1 -- 172.21.15.58:0/4203355429 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b7010caf0 con 0x7f8b700f6820 2021-11-20T05:38:42.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.600+0000 7f8b74b1c700 1 -- 172.21.15.58:0/4203355429 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b700fdd10 con 0x7f8b700f6820 2021-11-20T05:38:42.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.600+0000 7f8b5effd700 1 -- 172.21.15.58:0/4203355429 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8b64009480 con 0x7f8b700f6820 2021-11-20T05:38:42.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.601+0000 7f8b5effd700 1 --2- 172.21.15.58:0/4203355429 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b5805c2a0 0x7f8b5805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:42.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.601+0000 7f8b5effd700 1 -- 172.21.15.58:0/4203355429 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f8b64076a40 con 0x7f8b700f6820 2021-11-20T05:38:42.604 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.601+0000 7f8b6e59c700 1 --2- 172.21.15.58:0/4203355429 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b5805c2a0 0x7f8b5805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:42.604 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.601+0000 7f8b6e59c700 1 --2- 172.21.15.58:0/4203355429 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b5805c2a0 0x7f8b5805e760 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f8b60004e30 tx=0x7f8b60009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:42.605 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.605+0000 7f8b5effd700 1 -- 172.21.15.58:0/4203355429 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b64049280 con 0x7f8b700f6820 2021-11-20T05:38:42.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:42 smithi187 conmon[36358]: audit 2021-11-20T05:38:41.450212+0000 2021-11-20T05:38:42.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:42 smithi187 conmon[36358]: mon.smithi058 (mon.0) 870 : audit [DBG] from='client.? 172.21.15.58:0/810998476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:42.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:42 smithi058 conmon[33537]: audit 2021-11-20T05:38:41.450212+0000 mon.smithi058 (mon.0 2021-11-20T05:38:42.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:42 smithi058 conmon[33537]: ) 870 : audit [DBG] from='client.? 172.21.15.58:0/810998476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:42.798 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.803+0000 7f8b74b1c700 1 -- 172.21.15.58:0/4203355429 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b7010d240 con 0x7f8b700f6820 2021-11-20T05:38:42.799 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.803+0000 7f8b5effd700 1 -- 172.21.15.58:0/4203355429 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f8b6404c8a0 con 0x7f8b700f6820 2021-11-20T05:38:42.799 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:42.800 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:34.369456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24583,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24583":{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:42.801 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.805+0000 7f8b74b1c700 1 -- 172.21.15.58:0/4203355429 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b5805c2a0 msgr2=0x7f8b5805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:42.801 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.805+0000 7f8b74b1c700 1 --2- 172.21.15.58:0/4203355429 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b5805c2a0 0x7f8b5805e760 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f8b60004e30 tx=0x7f8b60009b30).stop 2021-11-20T05:38:42.801 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.805+0000 7f8b74b1c700 1 -- 172.21.15.58:0/4203355429 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b700f6820 msgr2=0x7f8b701073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:42.802 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.805+0000 7f8b74b1c700 1 --2- 172.21.15.58:0/4203355429 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b700f6820 0x7f8b701073a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f8b640026d0 tx=0x7f8b64009de0).stop 2021-11-20T05:38:42.802 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.805+0000 7f8b74b1c700 1 -- 172.21.15.58:0/4203355429 shutdown_connections 2021-11-20T05:38:42.802 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.805+0000 7f8b74b1c700 1 --2- 172.21.15.58:0/4203355429 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f8b5805c2a0 0x7f8b5805e760 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:42.802 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.805+0000 7f8b74b1c700 1 --2- 172.21.15.58:0/4203355429 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b700f5580 0x7f8b70106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:42.803 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.805+0000 7f8b74b1c700 1 --2- 172.21.15.58:0/4203355429 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f8b700f6820 0x7f8b701073a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:42.803 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.805+0000 7f8b74b1c700 1 -- 172.21.15.58:0/4203355429 >> 172.21.15.58:0/4203355429 conn(0x7f8b700f0ce0 msgr2=0x7f8b7005fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:42.803 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.805+0000 7f8b74b1c700 1 -- 172.21.15.58:0/4203355429 shutdown_connections 2021-11-20T05:38:42.803 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:42.805+0000 7f8b74b1c700 1 -- 172.21.15.58:0/4203355429 wait complete. 2021-11-20T05:38:42.804 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 96 2021-11-20T05:38:42.813 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:34.369456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24583, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 92, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6827/1745887865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 1745887865}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 1745887865}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:42.813 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T05:38:43.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:43 smithi187 conmon[36358]: cluster 2021-11-20T05:38:42.303906+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:38:43.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:43 smithi187 conmon[36358]: ) 330 : cluster [DBG] pgmap v307: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:38:43.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:43 smithi187 conmon[36358]: audit 2021-11-20T05:38:42.804575+0000 mon.smithi058 (mon.0) 871 : audit [DBG] 2021-11-20T05:38:43.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:43 smithi187 conmon[36358]: from='client.? 172.21.15.58:0/4203355429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:43.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:43 smithi058 conmon[33537]: cluster 2021-11-20T05:38:42.303906+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:38:43.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:43 smithi058 conmon[33537]: 330 : cluster [DBG] pgmap v307: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:38:43.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:43 smithi058 conmon[33537]: audit 2021-11-20T05:38:42.804575+0000 mon.smithi058 (mon.0) 871 : audit [DBG] from='client.? 172.21.15.58:0/4203355429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:43.814 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:43.948 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.951+0000 7fa1e06d7700 1 -- 172.21.15.58:0/3529747024 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1d80f9800 msgr2=0x7fa1d80fbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:43.949 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.951+0000 7fa1e06d7700 1 --2- 172.21.15.58:0/3529747024 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1d80f9800 0x7fa1d80fbc50 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7fa1cc004660 tx=0x7fa1cc009b30).stop 2021-11-20T05:38:43.949 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.951+0000 7fa1e06d7700 1 -- 172.21.15.58:0/3529747024 shutdown_connections 2021-11-20T05:38:43.949 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.951+0000 7fa1e06d7700 1 --2- 172.21.15.58:0/3529747024 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1d80f9800 0x7fa1d80fbc50 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:43.949 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.951+0000 7fa1e06d7700 1 --2- 172.21.15.58:0/3529747024 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1d8060140 0x7fa1d80f9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:43.950 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.951+0000 7fa1e06d7700 1 -- 172.21.15.58:0/3529747024 >> 172.21.15.58:0/3529747024 conn(0x7fa1d80f0ce0 msgr2=0x7fa1d80f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:43.950 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.951+0000 7fa1e06d7700 1 -- 172.21.15.58:0/3529747024 shutdown_connections 2021-11-20T05:38:43.950 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.951+0000 7fa1e06d7700 1 -- 172.21.15.58:0/3529747024 wait complete. 2021-11-20T05:38:43.950 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.951+0000 7fa1e06d7700 1 Processor -- start 2021-11-20T05:38:43.951 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.952+0000 7fa1e06d7700 1 -- start start 2021-11-20T05:38:43.951 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.952+0000 7fa1e06d7700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1d8060140 0x7fa1d80f6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:43.951 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.952+0000 7fa1e06d7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1d80f9800 0x7fa1d80f5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:43.951 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.952+0000 7fa1e06d7700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d80f70d0 con 0x7fa1d8060140 2021-11-20T05:38:43.951 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.952+0000 7fa1e06d7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d80f5670 con 0x7fa1d80f9800 2021-11-20T05:38:43.952 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.952+0000 7fa1de473700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1d8060140 0x7fa1d80f6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:43.952 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.952+0000 7fa1de473700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1d8060140 0x7fa1d80f6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42328/0 (socket says 172.21.15.58:42328) 2021-11-20T05:38:43.952 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.952+0000 7fa1de473700 1 -- 172.21.15.58:0/2200786635 learned_addr learned my addr 172.21.15.58:0/2200786635 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:43.952 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.952+0000 7fa1ddc72700 1 --2- 172.21.15.58:0/2200786635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1d80f9800 0x7fa1d80f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:43.953 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.952+0000 7fa1de473700 1 -- 172.21.15.58:0/2200786635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1d80f9800 msgr2=0x7fa1d80f5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:43.953 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.952+0000 7fa1de473700 1 --2- 172.21.15.58:0/2200786635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1d80f9800 0x7fa1d80f5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:43.953 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.952+0000 7fa1de473700 1 -- 172.21.15.58:0/2200786635 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1cc005040 con 0x7fa1d8060140 2021-11-20T05:38:43.954 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.952+0000 7fa1de473700 1 --2- 172.21.15.58:0/2200786635 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1d8060140 0x7fa1d80f6b20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7fa1d0007fd0 tx=0x7fa1d0004ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:43.954 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.952+0000 7fa1d6ffd700 1 -- 172.21.15.58:0/2200786635 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1d000b5a0 con 0x7fa1d8060140 2021-11-20T05:38:43.954 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.953+0000 7fa1d6ffd700 1 -- 172.21.15.58:0/2200786635 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1d000f040 con 0x7fa1d8060140 2021-11-20T05:38:43.954 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.953+0000 7fa1e06d7700 1 -- 172.21.15.58:0/2200786635 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d80f58f0 con 0x7fa1d8060140 2021-11-20T05:38:43.955 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.953+0000 7fa1d6ffd700 1 -- 172.21.15.58:0/2200786635 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1d0013b00 con 0x7fa1d8060140 2021-11-20T05:38:43.955 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.953+0000 7fa1e06d7700 1 -- 172.21.15.58:0/2200786635 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d80f5eb0 con 0x7fa1d8060140 2021-11-20T05:38:43.955 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.953+0000 7fa1e06d7700 1 -- 172.21.15.58:0/2200786635 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d8027a00 con 0x7fa1d8060140 2021-11-20T05:38:43.956 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.954+0000 7fa1d6ffd700 1 -- 172.21.15.58:0/2200786635 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa1d001a030 con 0x7fa1d8060140 2021-11-20T05:38:43.957 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.954+0000 7fa1d6ffd700 1 --2- 172.21.15.58:0/2200786635 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa1c005c2a0 0x7fa1c005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:43.957 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.954+0000 7fa1d6ffd700 1 -- 172.21.15.58:0/2200786635 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fa1d00761e0 con 0x7fa1d8060140 2021-11-20T05:38:43.957 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.954+0000 7fa1ddc72700 1 --2- 172.21.15.58:0/2200786635 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa1c005c2a0 0x7fa1c005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:43.957 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.955+0000 7fa1ddc72700 1 --2- 172.21.15.58:0/2200786635 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa1c005c2a0 0x7fa1c005e760 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fa1cc004e30 tx=0x7fa1cc009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:43.958 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:43.958+0000 7fa1d6ffd700 1 -- 172.21.15.58:0/2200786635 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1d0048a50 con 0x7fa1d8060140 2021-11-20T05:38:44.150 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:44.155+0000 7fa1e06d7700 1 -- 172.21.15.58:0/2200786635 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1d80fee30 con 0x7fa1d8060140 2021-11-20T05:38:44.151 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:44.155+0000 7fa1d6ffd700 1 -- 172.21.15.58:0/2200786635 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2575 (secure 0 0 0) 0x7fa1d0018070 con 0x7fa1d8060140 2021-11-20T05:38:44.151 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:44.152 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:34.369456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24583,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24583":{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:44.153 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:44.157+0000 7fa1e06d7700 1 -- 172.21.15.58:0/2200786635 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa1c005c2a0 msgr2=0x7fa1c005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:44.154 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:44.157+0000 7fa1e06d7700 1 --2- 172.21.15.58:0/2200786635 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa1c005c2a0 0x7fa1c005e760 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fa1cc004e30 tx=0x7fa1cc009b30).stop 2021-11-20T05:38:44.154 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:44.157+0000 7fa1e06d7700 1 -- 172.21.15.58:0/2200786635 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1d8060140 msgr2=0x7fa1d80f6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:44.154 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:44.157+0000 7fa1e06d7700 1 --2- 172.21.15.58:0/2200786635 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1d8060140 0x7fa1d80f6b20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7fa1d0007fd0 tx=0x7fa1d0004ee0).stop 2021-11-20T05:38:44.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:44.157+0000 7fa1e06d7700 1 -- 172.21.15.58:0/2200786635 shutdown_connections 2021-11-20T05:38:44.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:44.157+0000 7fa1e06d7700 1 --2- 172.21.15.58:0/2200786635 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa1c005c2a0 0x7fa1c005e760 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:44.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:44.157+0000 7fa1e06d7700 1 --2- 172.21.15.58:0/2200786635 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa1d8060140 0x7fa1d80f6b20 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:44.155 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:44.157+0000 7fa1e06d7700 1 --2- 172.21.15.58:0/2200786635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1d80f9800 0x7fa1d80f5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:44.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:44.157+0000 7fa1e06d7700 1 -- 172.21.15.58:0/2200786635 >> 172.21.15.58:0/2200786635 conn(0x7fa1d80f0ce0 msgr2=0x7fa1d80f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:44.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:44.157+0000 7fa1e06d7700 1 -- 172.21.15.58:0/2200786635 shutdown_connections 2021-11-20T05:38:44.156 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:44.157+0000 7fa1e06d7700 1 -- 172.21.15.58:0/2200786635 wait complete. 2021-11-20T05:38:44.156 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 96 2021-11-20T05:38:44.164 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:34.369456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24583, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 92, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6827/1745887865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 1745887865}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 1745887865}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:44.165 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T05:38:44.465 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:44 smithi058 conmon[33537]: audit 2021-11-20T05:38:44.156347+0000 mon.smithi058 (mon.0) 872 : audit [DBG] from='client.? 172.21.15.58:0/2200786635' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:44.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:44 smithi187 conmon[36358]: audit 2021-11-20T05:38:44.156347+0000 mon.smithi058 (mon.0) 2021-11-20T05:38:44.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:44 smithi187 conmon[36358]: 872 : audit [DBG] from='client.? 172.21.15.58:0/2200786635' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:45.166 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:45.301 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.304+0000 7f551c069700 1 -- 172.21.15.58:0/1079889078 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55140f7920 msgr2=0x7f55140f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:45.302 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.304+0000 7f551c069700 1 --2- 172.21.15.58:0/1079889078 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55140f7920 0x7f55140f7d80 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f5504004660 tx=0x7f5504009b30).stop 2021-11-20T05:38:45.302 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.304+0000 7f551c069700 1 -- 172.21.15.58:0/1079889078 shutdown_connections 2021-11-20T05:38:45.302 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.304+0000 7f551c069700 1 --2- 172.21.15.58:0/1079889078 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55140f7920 0x7f55140f7d80 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:45.302 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.304+0000 7f551c069700 1 --2- 172.21.15.58:0/1079889078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55140f6680 0x7f55140f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:45.303 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.304+0000 7f551c069700 1 -- 172.21.15.58:0/1079889078 >> 172.21.15.58:0/1079889078 conn(0x7f55140f1de0 msgr2=0x7f55140f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:45.303 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.304+0000 7f551c069700 1 -- 172.21.15.58:0/1079889078 shutdown_connections 2021-11-20T05:38:45.303 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.304+0000 7f551c069700 1 -- 172.21.15.58:0/1079889078 wait complete. 2021-11-20T05:38:45.303 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.304+0000 7f551c069700 1 Processor -- start 2021-11-20T05:38:45.303 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.304+0000 7f551c069700 1 -- start start 2021-11-20T05:38:45.304 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.304+0000 7f551c069700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55140f6680 0x7f551406b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:45.304 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.305+0000 7f551c069700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55140f7920 0x7f5514069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:45.304 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.305+0000 7f551c069700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551406bd70 con 0x7f55140f6680 2021-11-20T05:38:45.304 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.305+0000 7f551c069700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551406a310 con 0x7f55140f7920 2021-11-20T05:38:45.305 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.305+0000 7f5519e05700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55140f6680 0x7f551406b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:45.305 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.305+0000 7f5519e05700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55140f6680 0x7f551406b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42338/0 (socket says 172.21.15.58:42338) 2021-11-20T05:38:45.305 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.305+0000 7f5519e05700 1 -- 172.21.15.58:0/3114434134 learned_addr learned my addr 172.21.15.58:0/3114434134 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:45.305 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.305+0000 7f5519604700 1 --2- 172.21.15.58:0/3114434134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55140f7920 0x7f5514069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:45.305 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.305+0000 7f5519e05700 1 -- 172.21.15.58:0/3114434134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55140f7920 msgr2=0x7f5514069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:45.306 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.305+0000 7f5519e05700 1 --2- 172.21.15.58:0/3114434134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55140f7920 0x7f5514069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:45.306 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.305+0000 7f5519e05700 1 -- 172.21.15.58:0/3114434134 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5504005040 con 0x7f55140f6680 2021-11-20T05:38:45.306 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.305+0000 7f5519e05700 1 --2- 172.21.15.58:0/3114434134 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55140f6680 0x7f551406b7c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f5510002700 tx=0x7f5510009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:45.306 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.306+0000 7f550a7fc700 1 -- 172.21.15.58:0/3114434134 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55100079c0 con 0x7f55140f6680 2021-11-20T05:38:45.307 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.306+0000 7f551c069700 1 -- 172.21.15.58:0/3114434134 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551406a590 con 0x7f55140f6680 2021-11-20T05:38:45.307 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.306+0000 7f550a7fc700 1 -- 172.21.15.58:0/3114434134 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f551000f040 con 0x7f55140f6680 2021-11-20T05:38:45.307 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.306+0000 7f550a7fc700 1 -- 172.21.15.58:0/3114434134 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5510014860 con 0x7f55140f6680 2021-11-20T05:38:45.307 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.306+0000 7f551c069700 1 -- 172.21.15.58:0/3114434134 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f551406ab50 con 0x7f55140f6680 2021-11-20T05:38:45.309 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.307+0000 7f551c069700 1 -- 172.21.15.58:0/3114434134 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5514027a00 con 0x7f55140f6680 2021-11-20T05:38:45.309 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.307+0000 7f550a7fc700 1 -- 172.21.15.58:0/3114434134 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f551001b060 con 0x7f55140f6680 2021-11-20T05:38:45.309 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.308+0000 7f550a7fc700 1 --2- 172.21.15.58:0/3114434134 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f550005c2a0 0x7f550005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:45.311 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.308+0000 7f550a7fc700 1 -- 172.21.15.58:0/3114434134 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f5510076210 con 0x7f55140f6680 2021-11-20T05:38:45.311 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.308+0000 7f5519604700 1 --2- 172.21.15.58:0/3114434134 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f550005c2a0 0x7f550005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:45.311 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.308+0000 7f5519604700 1 --2- 172.21.15.58:0/3114434134 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f550005c2a0 0x7f550005e760 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f5504009f50 tx=0x7f5504009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:45.312 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.312+0000 7f550a7fc700 1 -- 172.21.15.58:0/3114434134 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f55100489d0 con 0x7f55140f6680 2021-11-20T05:38:45.465 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:45 smithi058 conmon[33537]: cluster 2021-11-20T05:38:44 2021-11-20T05:38:45.465 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:45 smithi058 conmon[33537]: .304420+0000 mgr.smithi058.zagdlk (mgr.14164) 331 : cluster [DBG] pgmap v308: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 0 op/s 2021-11-20T05:38:45.506 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.511+0000 7f551c069700 1 -- 172.21.15.58:0/3114434134 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55140fee10 con 0x7f55140f6680 2021-11-20T05:38:45.506 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.511+0000 7f550a7fc700 1 -- 172.21.15.58:0/3114434134 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f551004bff0 con 0x7f55140f6680 2021-11-20T05:38:45.507 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:45.507 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:34.369456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24583,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24583":{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:45.509 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.513+0000 7f551c069700 1 -- 172.21.15.58:0/3114434134 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f550005c2a0 msgr2=0x7f550005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:45.509 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.513+0000 7f551c069700 1 --2- 172.21.15.58:0/3114434134 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f550005c2a0 0x7f550005e760 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f5504009f50 tx=0x7f5504009b30).stop 2021-11-20T05:38:45.509 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.513+0000 7f551c069700 1 -- 172.21.15.58:0/3114434134 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55140f6680 msgr2=0x7f551406b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:45.509 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.513+0000 7f551c069700 1 --2- 172.21.15.58:0/3114434134 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55140f6680 0x7f551406b7c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f5510002700 tx=0x7f5510009d80).stop 2021-11-20T05:38:45.510 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.513+0000 7f551c069700 1 -- 172.21.15.58:0/3114434134 shutdown_connections 2021-11-20T05:38:45.510 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.513+0000 7f551c069700 1 --2- 172.21.15.58:0/3114434134 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f550005c2a0 0x7f550005e760 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:45.510 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.513+0000 7f551c069700 1 --2- 172.21.15.58:0/3114434134 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f55140f6680 0x7f551406b7c0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:45.510 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.513+0000 7f551c069700 1 --2- 172.21.15.58:0/3114434134 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55140f7920 0x7f5514069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:45.511 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.513+0000 7f551c069700 1 -- 172.21.15.58:0/3114434134 >> 172.21.15.58:0/3114434134 conn(0x7f55140f1de0 msgr2=0x7f55140fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:45.511 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.513+0000 7f551c069700 1 -- 172.21.15.58:0/3114434134 shutdown_connections 2021-11-20T05:38:45.511 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:45.513+0000 7f551c069700 1 -- 172.21.15.58:0/3114434134 wait complete. 2021-11-20T05:38:45.511 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 96 2021-11-20T05:38:45.520 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:34.369456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24583, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 92, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6827/1745887865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 1745887865}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 1745887865}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:45.521 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T05:38:45.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:45 smithi187 conmon[36358]: cluster 2021-11-20T05:38:44.304420+0000 mgr.smithi058.zagdlk (mgr.14164) 331 : cluster [DBG] pgmap v308: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 0 op/s 2021-11-20T05:38:46.522 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:46.657 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.659+0000 7fae0f7b0700 1 -- 172.21.15.58:0/387803440 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae080f6840 msgr2=0x7fae080f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:46.657 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.659+0000 7fae0f7b0700 1 --2- 172.21.15.58:0/387803440 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae080f6840 0x7fae080f6ca0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7fadf8004660 tx=0x7fadf8009b30).stop 2021-11-20T05:38:46.657 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.659+0000 7fae0f7b0700 1 -- 172.21.15.58:0/387803440 shutdown_connections 2021-11-20T05:38:46.657 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.659+0000 7fae0f7b0700 1 --2- 172.21.15.58:0/387803440 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae080f6840 0x7fae080f6ca0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:46.658 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.659+0000 7fae0f7b0700 1 --2- 172.21.15.58:0/387803440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae080f55a0 0x7fae080f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:46.658 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.659+0000 7fae0f7b0700 1 -- 172.21.15.58:0/387803440 >> 172.21.15.58:0/387803440 conn(0x7fae080f0ce0 msgr2=0x7fae080f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:46.658 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.659+0000 7fae0f7b0700 1 -- 172.21.15.58:0/387803440 shutdown_connections 2021-11-20T05:38:46.658 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.659+0000 7fae0f7b0700 1 -- 172.21.15.58:0/387803440 wait complete. 2021-11-20T05:38:46.659 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.660+0000 7fae0f7b0700 1 Processor -- start 2021-11-20T05:38:46.659 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.660+0000 7fae0f7b0700 1 -- start start 2021-11-20T05:38:46.659 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.660+0000 7fae0f7b0700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae080f55a0 0x7fae080f9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:46.659 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.660+0000 7fae0f7b0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae080f6840 0x7fae080f83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:46.660 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.660+0000 7fae0f7b0700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae080fa2f0 con 0x7fae080f55a0 2021-11-20T05:38:46.660 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.660+0000 7fae0f7b0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae080f8890 con 0x7fae080f6840 2021-11-20T05:38:46.660 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.660+0000 7fae0d54c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae080f55a0 0x7fae080f9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:46.660 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.660+0000 7fae0d54c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae080f55a0 0x7fae080f9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42348/0 (socket says 172.21.15.58:42348) 2021-11-20T05:38:46.660 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.660+0000 7fae0d54c700 1 -- 172.21.15.58:0/921473073 learned_addr learned my addr 172.21.15.58:0/921473073 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:46.661 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.660+0000 7fae0cd4b700 1 --2- 172.21.15.58:0/921473073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae080f6840 0x7fae080f83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:46.661 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.660+0000 7fae0d54c700 1 -- 172.21.15.58:0/921473073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae080f6840 msgr2=0x7fae080f83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:46.661 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.660+0000 7fae0d54c700 1 --2- 172.21.15.58:0/921473073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae080f6840 0x7fae080f83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:46.661 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.661+0000 7fae0d54c700 1 -- 172.21.15.58:0/921473073 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf8005040 con 0x7fae080f55a0 2021-11-20T05:38:46.662 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.661+0000 7fae0d54c700 1 --2- 172.21.15.58:0/921473073 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae080f55a0 0x7fae080f9d40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7fae0400c530 tx=0x7fae04007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:46.662 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.661+0000 7fadfdffb700 1 -- 172.21.15.58:0/921473073 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae0400f650 con 0x7fae080f55a0 2021-11-20T05:38:46.662 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.661+0000 7fadfdffb700 1 -- 172.21.15.58:0/921473073 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae0400fc60 con 0x7fae080f55a0 2021-11-20T05:38:46.662 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.661+0000 7fae0f7b0700 1 -- 172.21.15.58:0/921473073 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae080f8b10 con 0x7fae080f55a0 2021-11-20T05:38:46.662 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.661+0000 7fadfdffb700 1 -- 172.21.15.58:0/921473073 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae04012d20 con 0x7fae080f55a0 2021-11-20T05:38:46.663 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.661+0000 7fae0f7b0700 1 -- 172.21.15.58:0/921473073 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae080f90d0 con 0x7fae080f55a0 2021-11-20T05:38:46.664 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.662+0000 7fae0f7b0700 1 -- 172.21.15.58:0/921473073 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae080fdd30 con 0x7fae080f55a0 2021-11-20T05:38:46.664 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.662+0000 7fadfdffb700 1 -- 172.21.15.58:0/921473073 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fae04019030 con 0x7fae080f55a0 2021-11-20T05:38:46.664 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.663+0000 7fadfdffb700 1 --2- 172.21.15.58:0/921473073 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fadf405c2a0 0x7fadf405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:46.665 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.663+0000 7fadfdffb700 1 -- 172.21.15.58:0/921473073 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fae04075fc0 con 0x7fae080f55a0 2021-11-20T05:38:46.665 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.663+0000 7fae0cd4b700 1 --2- 172.21.15.58:0/921473073 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fadf405c2a0 0x7fadf405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:46.665 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.663+0000 7fae0cd4b700 1 --2- 172.21.15.58:0/921473073 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fadf405c2a0 0x7fadf405e760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fadf8013580 tx=0x7fadf8009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:46.665 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.667+0000 7fadfdffb700 1 -- 172.21.15.58:0/921473073 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae040462b0 con 0x7fae080f55a0 2021-11-20T05:38:46.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:46 smithi187 conmon[36358]: audit 2021-11-20T05:38:45.512116+0000 mon.smithi058 (mon.0) 873 : audit [DBG] from='client.? 172.21.15.58:0/3114434134' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:46.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:46 smithi058 conmon[33537]: audit 2021-11-20T05:38:45.512116+0000 mon.smithi058 (mon.0) 873 : audit [DBG] from='client.? 172.21.15.58:0/3114434134' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:46.856 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.861+0000 7fae0f7b0700 1 -- 172.21.15.58:0/921473073 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae0810f7f0 con 0x7fae080f55a0 2021-11-20T05:38:46.857 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.862+0000 7fadfdffb700 1 -- 172.21.15.58:0/921473073 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2575 (secure 0 0 0) 0x7fae04017020 con 0x7fae080f55a0 2021-11-20T05:38:46.858 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:46.858 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:34.369456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24583,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24583":{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:46.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.863+0000 7fae0f7b0700 1 -- 172.21.15.58:0/921473073 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fadf405c2a0 msgr2=0x7fadf405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:46.859 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.863+0000 7fae0f7b0700 1 --2- 172.21.15.58:0/921473073 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fadf405c2a0 0x7fadf405e760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fadf8013580 tx=0x7fadf8009b30).stop 2021-11-20T05:38:46.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.863+0000 7fae0f7b0700 1 -- 172.21.15.58:0/921473073 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae080f55a0 msgr2=0x7fae080f9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:46.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.863+0000 7fae0f7b0700 1 --2- 172.21.15.58:0/921473073 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae080f55a0 0x7fae080f9d40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7fae0400c530 tx=0x7fae04007e10).stop 2021-11-20T05:38:46.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.863+0000 7fae0f7b0700 1 -- 172.21.15.58:0/921473073 shutdown_connections 2021-11-20T05:38:46.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.863+0000 7fae0f7b0700 1 --2- 172.21.15.58:0/921473073 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fadf405c2a0 0x7fadf405e760 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:46.861 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.863+0000 7fae0f7b0700 1 --2- 172.21.15.58:0/921473073 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae080f55a0 0x7fae080f9d40 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:46.861 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.863+0000 7fae0f7b0700 1 --2- 172.21.15.58:0/921473073 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae080f6840 0x7fae080f83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:46.861 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.863+0000 7fae0f7b0700 1 -- 172.21.15.58:0/921473073 >> 172.21.15.58:0/921473073 conn(0x7fae080f0ce0 msgr2=0x7fae0805f450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:46.861 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.863+0000 7fae0f7b0700 1 -- 172.21.15.58:0/921473073 shutdown_connections 2021-11-20T05:38:46.862 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:46.863+0000 7fae0f7b0700 1 -- 172.21.15.58:0/921473073 wait complete. 2021-11-20T05:38:46.862 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 96 2021-11-20T05:38:46.870 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:34.369456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24583, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 92, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6827/1745887865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 1745887865}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 1745887865}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:46.871 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T05:38:47.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:47 smithi187 conmon[36358]: cluster 2021-11-20T05:38:46.304993+0000 mgr.smithi058.zagdlk (mgr.14164) 332 : cluster [DBG] pgmap v309: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:38:47.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:47 smithi187 conmon[36358]: audit 2021-11-20 2021-11-20T05:38:47.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:47 smithi187 conmon[36358]: T05:38:46.862628+0000 mon.smithi058 (mon.0) 874 : audit [DBG] from='client.? 172.21.15.58:0/921473073' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:47.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:47 smithi058 conmon[33537]: cluster 2021-11-20T05:38:46.304993+0000 mgr.smithi058.zagdlk (mgr.14164) 332 : cluster [DBG] pgmap v309: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:38:47.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:47 smithi058 conmon[33537]: audit 2021-11-20T05:38:46.862628+0000 mon.smithi058 (mon.0) 874 : audit [DBG] from='client.? 172.21.15.58:0/921473073' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:47.872 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:48.007 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.010+0000 7fcf86d37700 1 -- 172.21.15.58:0/4139428628 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcf800f55a0 msgr2=0x7fcf800f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:48.008 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.010+0000 7fcf86d37700 1 --2- 172.21.15.58:0/4139428628 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcf800f55a0 0x7fcf800f59c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7fcf780099e0 tx=0x7fcf78004d10).stop 2021-11-20T05:38:48.009 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.010+0000 7fcf86d37700 1 -- 172.21.15.58:0/4139428628 shutdown_connections 2021-11-20T05:38:48.009 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.010+0000 7fcf86d37700 1 --2- 172.21.15.58:0/4139428628 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf800f6840 0x7fcf800f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:48.009 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.010+0000 7fcf86d37700 1 --2- 172.21.15.58:0/4139428628 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcf800f55a0 0x7fcf800f59c0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:48.009 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.010+0000 7fcf86d37700 1 -- 172.21.15.58:0/4139428628 >> 172.21.15.58:0/4139428628 conn(0x7fcf800f0ce0 msgr2=0x7fcf800f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:48.010 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.010+0000 7fcf86d37700 1 -- 172.21.15.58:0/4139428628 shutdown_connections 2021-11-20T05:38:48.010 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.010+0000 7fcf86d37700 1 -- 172.21.15.58:0/4139428628 wait complete. 2021-11-20T05:38:48.010 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.010+0000 7fcf86d37700 1 Processor -- start 2021-11-20T05:38:48.010 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.010+0000 7fcf86d37700 1 -- start start 2021-11-20T05:38:48.011 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.011+0000 7fcf86d37700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcf800f55a0 0x7fcf80106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:48.011 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.011+0000 7fcf86d37700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf800f6840 0x7fcf801074a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:48.011 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.011+0000 7fcf86d37700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80107a50 con 0x7fcf800f55a0 2021-11-20T05:38:48.011 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.011+0000 7fcf86d37700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80107b90 con 0x7fcf800f6840 2021-11-20T05:38:48.011 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.011+0000 7fcf84ad3700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcf800f55a0 0x7fcf80106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:48.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.011+0000 7fcf84ad3700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcf800f55a0 0x7fcf80106fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42360/0 (socket says 172.21.15.58:42360) 2021-11-20T05:38:48.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.011+0000 7fcf84ad3700 1 -- 172.21.15.58:0/158053056 learned_addr learned my addr 172.21.15.58:0/158053056 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:48.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.011+0000 7fcf7ffff700 1 --2- 172.21.15.58:0/158053056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf800f6840 0x7fcf801074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:48.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.011+0000 7fcf84ad3700 1 -- 172.21.15.58:0/158053056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf800f6840 msgr2=0x7fcf801074a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:48.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.011+0000 7fcf84ad3700 1 --2- 172.21.15.58:0/158053056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf800f6840 0x7fcf801074a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:48.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.011+0000 7fcf84ad3700 1 -- 172.21.15.58:0/158053056 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf78009640 con 0x7fcf800f55a0 2021-11-20T05:38:48.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.011+0000 7fcf7ffff700 1 --2- 172.21.15.58:0/158053056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf800f6840 0x7fcf801074a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T05:38:48.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.011+0000 7fcf84ad3700 1 --2- 172.21.15.58:0/158053056 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcf800f55a0 0x7fcf80106fb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7fcf780078b0 tx=0x7fcf78004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:48.014 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.011+0000 7fcf7d7fa700 1 -- 172.21.15.58:0/158053056 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf7800e030 con 0x7fcf800f55a0 2021-11-20T05:38:48.014 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.012+0000 7fcf7d7fa700 1 -- 172.21.15.58:0/158053056 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf78007e00 con 0x7fcf800f55a0 2021-11-20T05:38:48.015 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.012+0000 7fcf86d37700 1 -- 172.21.15.58:0/158053056 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf8010c5d0 con 0x7fcf800f55a0 2021-11-20T05:38:48.015 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.012+0000 7fcf7d7fa700 1 -- 172.21.15.58:0/158053056 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf78018070 con 0x7fcf800f55a0 2021-11-20T05:38:48.015 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.012+0000 7fcf86d37700 1 -- 172.21.15.58:0/158053056 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf8010cb10 con 0x7fcf800f55a0 2021-11-20T05:38:48.015 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.012+0000 7fcf86d37700 1 -- 172.21.15.58:0/158053056 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf800fdd50 con 0x7fcf800f55a0 2021-11-20T05:38:48.016 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.013+0000 7fcf7d7fa700 1 -- 172.21.15.58:0/158053056 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcf78013790 con 0x7fcf800f55a0 2021-11-20T05:38:48.017 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.013+0000 7fcf7d7fa700 1 --2- 172.21.15.58:0/158053056 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcf6805c2a0 0x7fcf6805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:48.017 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.013+0000 7fcf7d7fa700 1 -- 172.21.15.58:0/158053056 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fcf78078110 con 0x7fcf800f55a0 2021-11-20T05:38:48.017 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.014+0000 7fcf7ffff700 1 --2- 172.21.15.58:0/158053056 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcf6805c2a0 0x7fcf6805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:48.017 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.014+0000 7fcf7ffff700 1 --2- 172.21.15.58:0/158053056 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcf6805c2a0 0x7fcf6805e760 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fcf74009740 tx=0x7fcf74009c20).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:48.018 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.017+0000 7fcf7d7fa700 1 -- 172.21.15.58:0/158053056 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf7804a950 con 0x7fcf800f55a0 2021-11-20T05:38:48.213 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.218+0000 7fcf86d37700 1 -- 172.21.15.58:0/158053056 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf8010d580 con 0x7fcf800f55a0 2021-11-20T05:38:48.214 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.219+0000 7fcf7d7fa700 1 -- 172.21.15.58:0/158053056 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2575 (secure 0 0 0) 0x7fcf7804df70 con 0x7fcf800f55a0 2021-11-20T05:38:48.215 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:48.215 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":96}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:34.369456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24583,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24583":{"gid":24583,"name":"cephfs.smithi187.fikisa","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.187:6827/1745887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1745887865},{"type":"v1","addr":"172.21.15.187:6827","nonce":1745887865}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:48.216 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.220+0000 7fcf86d37700 1 -- 172.21.15.58:0/158053056 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcf6805c2a0 msgr2=0x7fcf6805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:48.217 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.220+0000 7fcf86d37700 1 --2- 172.21.15.58:0/158053056 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcf6805c2a0 0x7fcf6805e760 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fcf74009740 tx=0x7fcf74009c20).stop 2021-11-20T05:38:48.217 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.220+0000 7fcf86d37700 1 -- 172.21.15.58:0/158053056 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcf800f55a0 msgr2=0x7fcf80106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:48.217 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.220+0000 7fcf86d37700 1 --2- 172.21.15.58:0/158053056 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcf800f55a0 0x7fcf80106fb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7fcf780078b0 tx=0x7fcf78004d10).stop 2021-11-20T05:38:48.217 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.220+0000 7fcf86d37700 1 -- 172.21.15.58:0/158053056 shutdown_connections 2021-11-20T05:38:48.217 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.220+0000 7fcf86d37700 1 --2- 172.21.15.58:0/158053056 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fcf6805c2a0 0x7fcf6805e760 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:48.218 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.220+0000 7fcf86d37700 1 --2- 172.21.15.58:0/158053056 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fcf800f55a0 0x7fcf80106fb0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:48.218 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.220+0000 7fcf86d37700 1 --2- 172.21.15.58:0/158053056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf800f6840 0x7fcf801074a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:48.218 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.220+0000 7fcf86d37700 1 -- 172.21.15.58:0/158053056 >> 172.21.15.58:0/158053056 conn(0x7fcf800f0ce0 msgr2=0x7fcf8005f450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:48.218 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.220+0000 7fcf86d37700 1 -- 172.21.15.58:0/158053056 shutdown_connections 2021-11-20T05:38:48.219 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:48.220+0000 7fcf86d37700 1 -- 172.21.15.58:0/158053056 wait complete. 2021-11-20T05:38:48.219 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 96 2021-11-20T05:38:48.227 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:34.369456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24583, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi187.fikisa', 'rank': 0, 'incarnation': 92, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.187:6827/1745887865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 1745887865}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 1745887865}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:48.228 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T05:38:48.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:48 smithi187 conmon[36358]: audit 2021-11-20T05:38:48.219643+0000 mon.smithi058 (mon.0) 875 : audit [DBG] from='client.? 172.21.15.58:0/158053056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:48.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:48 smithi058 conmon[33537]: audit 2021-11-20T05:38:48.219643+0000 mon.smithi058 2021-11-20T05:38:48.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:48 smithi058 conmon[33537]: (mon.0) 875 : audit [DBG] from='client.? 172.21.15.58:0/158053056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:49.229 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:49.369 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.371+0000 7f90f527f700 1 -- 172.21.15.58:0/1420332788 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f90f00f5f30 msgr2=0x7f90f00f8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:49.370 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.371+0000 7f90f527f700 1 --2- 172.21.15.58:0/1420332788 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f90f00f5f30 0x7f90f00f8320 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f90e4004660 tx=0x7f90e4009b30).stop 2021-11-20T05:38:49.370 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.371+0000 7f90f527f700 1 -- 172.21.15.58:0/1420332788 shutdown_connections 2021-11-20T05:38:49.370 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.371+0000 7f90f527f700 1 --2- 172.21.15.58:0/1420332788 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90f00f8810 0x7f90f00fac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:49.370 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.371+0000 7f90f527f700 1 --2- 172.21.15.58:0/1420332788 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f90f00f5f30 0x7f90f00f8320 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:49.371 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.371+0000 7f90f527f700 1 -- 172.21.15.58:0/1420332788 >> 172.21.15.58:0/1420332788 conn(0x7f90f00efff0 msgr2=0x7f90f00f2430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:49.371 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.371+0000 7f90f527f700 1 -- 172.21.15.58:0/1420332788 shutdown_connections 2021-11-20T05:38:49.371 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.371+0000 7f90f527f700 1 -- 172.21.15.58:0/1420332788 wait complete. 2021-11-20T05:38:49.371 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.372+0000 7f90f527f700 1 Processor -- start 2021-11-20T05:38:49.371 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.372+0000 7f90f527f700 1 -- start start 2021-11-20T05:38:49.372 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.372+0000 7f90f527f700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f90f00f5f30 0x7f90f00f5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:49.372 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.372+0000 7f90f527f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90f00f8810 0x7f90f010e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:49.372 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.372+0000 7f90f527f700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f0107aa0 con 0x7f90f00f5f30 2021-11-20T05:38:49.372 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.372+0000 7f90f527f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f010e5b0 con 0x7f90f00f8810 2021-11-20T05:38:49.373 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.372+0000 7f90eeffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f90f00f5f30 0x7f90f00f5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:49.373 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.372+0000 7f90eeffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f90f00f5f30 0x7f90f00f5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42370/0 (socket says 172.21.15.58:42370) 2021-11-20T05:38:49.373 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.372+0000 7f90eeffd700 1 -- 172.21.15.58:0/2315074648 learned_addr learned my addr 172.21.15.58:0/2315074648 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:49.373 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.372+0000 7f90ee7fc700 1 --2- 172.21.15.58:0/2315074648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90f00f8810 0x7f90f010e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:49.373 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.373+0000 7f90eeffd700 1 -- 172.21.15.58:0/2315074648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90f00f8810 msgr2=0x7f90f010e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:49.374 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.373+0000 7f90eeffd700 1 --2- 172.21.15.58:0/2315074648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90f00f8810 0x7f90f010e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:49.374 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.373+0000 7f90eeffd700 1 -- 172.21.15.58:0/2315074648 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e4005040 con 0x7f90f00f5f30 2021-11-20T05:38:49.374 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.373+0000 7f90eeffd700 1 --2- 172.21.15.58:0/2315074648 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f90f00f5f30 0x7f90f00f5710 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f90e4009f50 tx=0x7f90e4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:49.374 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.373+0000 7f90d77fe700 1 -- 172.21.15.58:0/2315074648 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90e4018070 con 0x7f90f00f5f30 2021-11-20T05:38:49.375 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.373+0000 7f90d77fe700 1 -- 172.21.15.58:0/2315074648 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90e4013410 con 0x7f90f00f5f30 2021-11-20T05:38:49.376 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.373+0000 7f90f527f700 1 -- 172.21.15.58:0/2315074648 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f010e6f0 con 0x7f90f00f5f30 2021-11-20T05:38:49.376 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.373+0000 7f90d77fe700 1 -- 172.21.15.58:0/2315074648 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90e401de30 con 0x7f90f00f5f30 2021-11-20T05:38:49.376 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.373+0000 7f90f527f700 1 -- 172.21.15.58:0/2315074648 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f010ec20 con 0x7f90f00f5f30 2021-11-20T05:38:49.376 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.374+0000 7f90f527f700 1 -- 172.21.15.58:0/2315074648 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f0046330 con 0x7f90f00f5f30 2021-11-20T05:38:49.378 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.374+0000 7f90d77fe700 1 -- 172.21.15.58:0/2315074648 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f90e401d420 con 0x7f90f00f5f30 2021-11-20T05:38:49.378 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.375+0000 7f90d77fe700 1 --2- 172.21.15.58:0/2315074648 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f90d805c240 0x7f90d805e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:49.378 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.375+0000 7f90d77fe700 1 -- 172.21.15.58:0/2315074648 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f90e4077fc0 con 0x7f90f00f5f30 2021-11-20T05:38:49.378 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.375+0000 7f90ee7fc700 1 --2- 172.21.15.58:0/2315074648 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f90d805c240 0x7f90d805e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:49.379 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.375+0000 7f90ee7fc700 1 --2- 172.21.15.58:0/2315074648 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f90d805c240 0x7f90d805e700 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f90f00f4c30 tx=0x7f90e000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:49.379 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.379+0000 7f90d77fe700 1 -- 172.21.15.58:0/2315074648 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f90e404a720 con 0x7f90f00f5f30 2021-11-20T05:38:49.572 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.576+0000 7f90f527f700 1 -- 172.21.15.58:0/2315074648 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90f00fddc0 con 0x7f90f00f5f30 2021-11-20T05:38:49.572 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.577+0000 7f90d77fe700 1 -- 172.21.15.58:0/2315074648 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+2198 (secure 0 0 0) 0x7f90e404dd40 con 0x7f90f00f5f30 2021-11-20T05:38:49.573 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:49.574 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:49.276453+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:replay","state_seq":3,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:49.575 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.578+0000 7f90f527f700 1 -- 172.21.15.58:0/2315074648 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f90d805c240 msgr2=0x7f90d805e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:49.575 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.578+0000 7f90f527f700 1 --2- 172.21.15.58:0/2315074648 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f90d805c240 0x7f90d805e700 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f90f00f4c30 tx=0x7f90e000b040).stop 2021-11-20T05:38:49.576 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.578+0000 7f90f527f700 1 -- 172.21.15.58:0/2315074648 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f90f00f5f30 msgr2=0x7f90f00f5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:49.576 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.578+0000 7f90f527f700 1 --2- 172.21.15.58:0/2315074648 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f90f00f5f30 0x7f90f00f5710 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f90e4009f50 tx=0x7f90e4009b30).stop 2021-11-20T05:38:49.576 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.578+0000 7f90f527f700 1 -- 172.21.15.58:0/2315074648 shutdown_connections 2021-11-20T05:38:49.576 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.578+0000 7f90f527f700 1 --2- 172.21.15.58:0/2315074648 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f90d805c240 0x7f90d805e700 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:49.577 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.578+0000 7f90f527f700 1 --2- 172.21.15.58:0/2315074648 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f90f00f5f30 0x7f90f00f5710 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:49.577 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.578+0000 7f90f527f700 1 --2- 172.21.15.58:0/2315074648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f90f00f8810 0x7f90f010e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:49.577 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.578+0000 7f90f527f700 1 -- 172.21.15.58:0/2315074648 >> 172.21.15.58:0/2315074648 conn(0x7f90f00efff0 msgr2=0x7f90f00f2430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:49.577 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.578+0000 7f90f527f700 1 -- 172.21.15.58:0/2315074648 shutdown_connections 2021-11-20T05:38:49.578 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:49.578+0000 7f90f527f700 1 -- 172.21.15.58:0/2315074648 wait complete. 2021-11-20T05:38:49.578 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 97 2021-11-20T05:38:49.584 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:49 smithi058 conmon[33537]: cluster 2021-11-20T05:38:48.305537+0000 mgr.smithi058.zagdlk 2021-11-20T05:38:49.584 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:49 smithi058 conmon[33537]: (mgr.14164) 333 : cluster [DBG] pgmap v310: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:38:49.585 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:49 smithi058 conmon[33537]: cluster 2021-11-20T05:38:49.275859+0000 mon.smithi058 (mon.0) 876 : cluster [WRN] Replacing daemon mds.cephfs.smithi187.fikisa as rank 0 with standby daemon mds.cephfs.smithi187.wxduhu 2021-11-20T05:38:49.585 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:49 smithi058 conmon[33537]: cluster 2021-11-20T05:38:49.275934+0000 mon.smithi058 (mon.0) 877 : cluster [INF] MDS daemon mds.cephfs.smithi187.fikisa is removed because it is dead or otherwise unavailable. 2021-11-20T05:38:49.586 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:49 smithi058 conmon[33537]: cluster 2021-11-20T05:38:49.276558+0000 mon.smithi058 ( 2021-11-20T05:38:49.586 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:49 smithi058 conmon[33537]: mon.0) 878 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2021-11-20T05:38:49.586 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:49 smithi058 conmon[33537]: cluster 2021-11-20T05: 2021-11-20T05:38:49.586 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:49 smithi058 conmon[33537]: 38:49.280779+0000 mon.smithi058 (mon.0) 879 : cluster 2021-11-20T05:38:49.587 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:49 smithi058 conmon[33537]: [DBG] osdmap e58: 6 total, 6 up, 6 in 2021-11-20T05:38:49.587 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:49 smithi058 conmon[33537]: cluster 2021-11-20T05:38:49.587 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:49 smithi058 conmon[33537]: 2021-11-20T05:38:49.284437+0000 2021-11-20T05:38:49.587 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:49 smithi058 conmon[33537]: mon.smithi058 (mon.0) 880 : cluster [DBG] 2021-11-20T05:38:49.588 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:49 smithi058 conmon[33537]: fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2021-11-20T05:38:49.589 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:49.276453+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:replay', 'state_seq': 3, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:49.589 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:replay 2021-11-20T05:38:49.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:49 smithi187 conmon[36358]: cluster 2021-11-20T05:38:48.305537+0000 mgr.smithi058.zagdlk (mgr.14164) 333 : cluster [DBG] pgmap v310: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:38:49.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:49 smithi187 conmon[36358]: cluster 2021-11-20T05:38:49.275859+0000 mon.smithi058 2021-11-20T05:38:49.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:49 smithi187 conmon[36358]: (mon.0) 876 : cluster [WRN] Replacing daemon mds.cephfs.smithi187.fikisa as rank 0 with standby daemon mds.cephfs.smithi187.wxduhu 2021-11-20T05:38:49.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:49 smithi187 conmon[36358]: cluster 2021-11-20T05:38:49.275934+0000 mon.smithi058 (mon.0) 877 : cluster [INF] MDS daemon mds.cephfs.smithi187.fikisa is removed because it is dead or otherwise unavailable. 2021-11-20T05:38:49.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:49 smithi187 conmon[36358]: cluster 2021-11-20T05:38:49.276558+0000 mon.smithi058 (mon.0) 878 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2021-11-20T05:38:49.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:49 smithi187 conmon[36358]: cluster 2021-11-20T05:38:49.280779+0000 mon.smithi058 (mon.0) 879 : cluster 2021-11-20T05:38:49.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:49 smithi187 conmon[36358]: [DBG] osdmap e58: 6 total, 6 up, 6 in 2021-11-20T05:38:49.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:49 smithi187 conmon[36358]: cluster 2021-11-20T05: 2021-11-20T05:38:49.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:49 smithi187 conmon[36358]: 38:49.284437+0000 mon.smithi058 (mon.0) 880 2021-11-20T05:38:49.695 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:49 smithi187 conmon[36358]: : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:replay,1=cephfs.smithi058.nuxpgf=up:active} 2021-11-20T05:38:50.590 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:50.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:50 smithi187 conmon[36358]: audit 2021-11-20T05:38:49.577747+0000 mon.smithi058 (mon.0) 881 : audit [DBG] from='client.? 172.21.15.58:0/2315074648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:50.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:50 smithi187 conmon[36358]: cluster 2021-11-20T05:38:50.286131+0000 mon.smithi058 (mon.0) 882 : cluster [DBG] mds.? [v2:172.21.15.187:6824/1414418853,v1:172.21.15.187:6825/1414418853] up:resolve 2021-11-20T05:38:50.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:50 smithi187 conmon[36358]: cluster 2021-11-20T05:38:50.286206+0000 mon.smithi058 (mon.0) 883 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2021-11-20T05:38:50.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:50 smithi058 conmon[33537]: audit 2021-11-20T05:38:49.577747+0000 mon.smithi058 (mon.0) 881 : audit [DBG] from='client.? 172.21.15.58:0/2315074648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:50.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:50 smithi058 conmon[33537]: cluster 2021-11-20T05:38:50.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:50 smithi058 conmon[33537]: 2021-11-20T05:38:50.286131+0000 mon.smithi058 (mon.0) 882 : cluster [DBG] mds.? [v2:172.21.15.187:6824/1414418853,v1:172.21.15.187:6825/1414418853] up:resolve 2021-11-20T05:38:50.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:50 smithi058 conmon[33537]: cluster 2021-11-20T05: 2021-11-20T05:38:50.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:50 smithi058 conmon[33537]: 38:50.286206+0000 mon.smithi058 (mon.0) 883 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:resolve,1=cephfs.smithi058.nuxpgf=up:active} 2021-11-20T05:38:50.725 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.727+0000 7f8807253700 1 -- 172.21.15.58:0/1054436272 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f88000f4960 msgr2=0x7f88000f4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:50.725 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.727+0000 7f8807253700 1 --2- 172.21.15.58:0/1054436272 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f88000f4960 0x7f88000f4d80 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f87f4004660 tx=0x7f87f4009b30).stop 2021-11-20T05:38:50.726 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.728+0000 7f8807253700 1 -- 172.21.15.58:0/1054436272 shutdown_connections 2021-11-20T05:38:50.726 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.728+0000 7f8807253700 1 --2- 172.21.15.58:0/1054436272 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88000f5300 0x7f88000f7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:50.726 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.728+0000 7f8807253700 1 --2- 172.21.15.58:0/1054436272 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f88000f4960 0x7f88000f4d80 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:50.726 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.728+0000 7f8807253700 1 -- 172.21.15.58:0/1054436272 >> 172.21.15.58:0/1054436272 conn(0x7f88000f0440 msgr2=0x7f88000f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:50.727 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.728+0000 7f8807253700 1 -- 172.21.15.58:0/1054436272 shutdown_connections 2021-11-20T05:38:50.727 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.728+0000 7f8807253700 1 -- 172.21.15.58:0/1054436272 wait complete. 2021-11-20T05:38:50.727 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.728+0000 7f8807253700 1 Processor -- start 2021-11-20T05:38:50.727 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.728+0000 7f8807253700 1 -- start start 2021-11-20T05:38:50.729 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.728+0000 7f8807253700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f88000f4960 0x7f8800106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:50.730 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.729+0000 7f8807253700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88000f5300 0x7f88001073e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:50.730 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.729+0000 7f8807253700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8800107990 con 0x7f88000f4960 2021-11-20T05:38:50.730 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.729+0000 7f8807253700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8800107ad0 con 0x7f88000f5300 2021-11-20T05:38:50.730 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.729+0000 7f8804fef700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f88000f4960 0x7f8800106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:50.731 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.729+0000 7f8804fef700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f88000f4960 0x7f8800106ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42380/0 (socket says 172.21.15.58:42380) 2021-11-20T05:38:50.731 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.729+0000 7f8804fef700 1 -- 172.21.15.58:0/1084603289 learned_addr learned my addr 172.21.15.58:0/1084603289 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:50.731 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.729+0000 7f8804fef700 1 -- 172.21.15.58:0/1084603289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88000f5300 msgr2=0x7f88001073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:50.731 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.729+0000 7f8804fef700 1 --2- 172.21.15.58:0/1084603289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88000f5300 0x7f88001073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:38:50.731 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.729+0000 7f8804fef700 1 -- 172.21.15.58:0/1084603289 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f4005040 con 0x7f88000f4960 2021-11-20T05:38:50.732 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.729+0000 7f8804fef700 1 --2- 172.21.15.58:0/1084603289 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f88000f4960 0x7f8800106ef0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f87f4004e30 tx=0x7f87f4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:50.732 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.729+0000 7f87fd7fa700 1 -- 172.21.15.58:0/1084603289 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87f4018070 con 0x7f88000f4960 2021-11-20T05:38:50.732 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.729+0000 7f87fd7fa700 1 -- 172.21.15.58:0/1084603289 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f87f4013410 con 0x7f88000f4960 2021-11-20T05:38:50.732 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.729+0000 7f8807253700 1 -- 172.21.15.58:0/1084603289 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f880010c510 con 0x7f88000f4960 2021-11-20T05:38:50.733 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.729+0000 7f87fd7fa700 1 -- 172.21.15.58:0/1084603289 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87f401c3c0 con 0x7f88000f4960 2021-11-20T05:38:50.733 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.729+0000 7f8807253700 1 -- 172.21.15.58:0/1084603289 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f880010cad0 con 0x7f88000f4960 2021-11-20T05:38:50.733 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.730+0000 7f8807253700 1 -- 172.21.15.58:0/1084603289 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8800027a00 con 0x7f88000f4960 2021-11-20T05:38:50.734 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.730+0000 7f87fd7fa700 1 -- 172.21.15.58:0/1084603289 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f87f401d420 con 0x7f88000f4960 2021-11-20T05:38:50.735 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.731+0000 7f87fd7fa700 1 --2- 172.21.15.58:0/1084603289 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f87f005c2a0 0x7f87f005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:50.735 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.731+0000 7f87fd7fa700 1 -- 172.21.15.58:0/1084603289 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f87f4077ff0 con 0x7f88000f4960 2021-11-20T05:38:50.736 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.731+0000 7f87fffff700 1 --2- 172.21.15.58:0/1084603289 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f87f005c2a0 0x7f87f005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:50.736 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.731+0000 7f87fffff700 1 --2- 172.21.15.58:0/1084603289 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f87f005c2a0 0x7f87f005e760 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f88000f5740 tx=0x7f87ec0099e0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:50.736 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.735+0000 7f87fd7fa700 1 -- 172.21.15.58:0/1084603289 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87f404a7d0 con 0x7f88000f4960 2021-11-20T05:38:50.924 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.928+0000 7f8807253700 1 -- 172.21.15.58:0/1084603289 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88000fee70 con 0x7f88000f4960 2021-11-20T05:38:50.924 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.929+0000 7f87fd7fa700 1 -- 172.21.15.58:0/1084603289 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+2199 (secure 0 0 0) 0x7f87f404ddf0 con 0x7f88000f4960 2021-11-20T05:38:50.925 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:50.925 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:50.281933+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:resolve","state_seq":7,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:50.926 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.931+0000 7f8807253700 1 -- 172.21.15.58:0/1084603289 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f87f005c2a0 msgr2=0x7f87f005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:50.926 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.931+0000 7f8807253700 1 --2- 172.21.15.58:0/1084603289 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f87f005c2a0 0x7f87f005e760 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f88000f5740 tx=0x7f87ec0099e0).stop 2021-11-20T05:38:50.927 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.931+0000 7f8807253700 1 -- 172.21.15.58:0/1084603289 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f88000f4960 msgr2=0x7f8800106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:50.927 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.931+0000 7f8807253700 1 --2- 172.21.15.58:0/1084603289 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f88000f4960 0x7f8800106ef0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f87f4004e30 tx=0x7f87f4009b30).stop 2021-11-20T05:38:50.927 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.931+0000 7f8807253700 1 -- 172.21.15.58:0/1084603289 shutdown_connections 2021-11-20T05:38:50.927 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.931+0000 7f8807253700 1 --2- 172.21.15.58:0/1084603289 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f87f005c2a0 0x7f87f005e760 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:50.928 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.931+0000 7f8807253700 1 --2- 172.21.15.58:0/1084603289 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f88000f4960 0x7f8800106ef0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:50.928 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.931+0000 7f8807253700 1 --2- 172.21.15.58:0/1084603289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88000f5300 0x7f88001073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:38:50.928 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.931+0000 7f8807253700 1 -- 172.21.15.58:0/1084603289 >> 172.21.15.58:0/1084603289 conn(0x7f88000f0440 msgr2=0x7f88000fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:50.928 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.931+0000 7f8807253700 1 -- 172.21.15.58:0/1084603289 shutdown_connections 2021-11-20T05:38:50.929 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:50.931+0000 7f8807253700 1 -- 172.21.15.58:0/1084603289 wait complete. 2021-11-20T05:38:50.929 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 98 2021-11-20T05:38:50.938 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:50.281933+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:50.938 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:resolve 2021-11-20T05:38:51.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:51 smithi187 conmon[36358]: cluster 2021-11-20T05:38:50.306020+0000 mgr.smithi058.zagdlk (mgr.14164) 334 : cluster [DBG] pgmap v312: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:38:51.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:51 smithi187 conmon[36358]: audit 2021-11-20T05:38:50.930024+0000 mon.smithi058 (mon.0) 884 : audit [DBG] from='client.? 172.21.15.58:0/1084603289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:51.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:51 smithi187 conmon[36358]: cluster 2021-11-20T05:38:51.289604+0000 mon.smithi058 (mon.0) 885 : cluster [DBG] mds.? [v2:172.21.15.187:6824/1414418853,v1:172.21.15.187:6825/1414418853] up:reconnect 2021-11-20T05:38:51.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:51 smithi187 conmon[36358]: cluster 2021-11-20T05:38:51.289675+0000 2021-11-20T05:38:51.694 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:51 smithi187 conmon[36358]: mon.smithi058 (mon.0) 886 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2021-11-20T05:38:51.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:51 smithi058 conmon[33537]: cluster 2021-11-20T05:38:50.306020+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:38:51.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:51 smithi058 conmon[33537]: ) 334 : cluster [DBG] pgmap v312: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:38:51.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:51 smithi058 conmon[33537]: audit 2021-11-20T05:38:50.930024+0000 mon.smithi058 (mon.0) 884 : audit [DBG] 2021-11-20T05:38:51.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:51 smithi058 conmon[33537]: from='client.? 172.21.15.58:0/1084603289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:51.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:51 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:38:51.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:51 smithi058 conmon[33537]: -11-20T05:38:51.289604+0000 2021-11-20T05:38:51.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:51 smithi058 conmon[33537]: mon.smithi058 (mon.0) 885 : cluster [DBG] mds.? [v2:172.21.15.187:6824/1414418853,v1:172.21.15.187:6825/1414418853] up:reconnect 2021-11-20T05:38:51.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:51 smithi058 conmon[33537]: cluster 2021 2021-11-20T05:38:51.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:51 smithi058 conmon[33537]: -11-20T05:38:51 2021-11-20T05:38:51.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:51 smithi058 conmon[33537]: .289675+0000 mon.smithi058 (mon.0 2021-11-20T05:38:51.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:51 smithi058 conmon[33537]: ) 886 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:reconnect,1=cephfs.smithi058.nuxpgf=up:active} 2021-11-20T05:38:51.718 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:51 smithi058 conmon[33537]: 2021-11-20T05:38:51.939 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:52.073 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.076+0000 7f71c1a90700 1 -- 172.21.15.58:0/2573318356 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f71bc0f7920 msgr2=0x7f71bc0f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:52.074 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.076+0000 7f71c1a90700 1 --2- 172.21.15.58:0/2573318356 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f71bc0f7920 0x7f71bc0f7d80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f71b0004660 tx=0x7f71b0009b30).stop 2021-11-20T05:38:52.074 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.076+0000 7f71c1a90700 1 -- 172.21.15.58:0/2573318356 shutdown_connections 2021-11-20T05:38:52.074 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.076+0000 7f71c1a90700 1 --2- 172.21.15.58:0/2573318356 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f71bc0f7920 0x7f71bc0f7d80 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:52.075 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.076+0000 7f71c1a90700 1 --2- 172.21.15.58:0/2573318356 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71bc0f6680 0x7f71bc0f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:52.075 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.076+0000 7f71c1a90700 1 -- 172.21.15.58:0/2573318356 >> 172.21.15.58:0/2573318356 conn(0x7f71bc0f1de0 msgr2=0x7f71bc0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:52.075 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.076+0000 7f71c1a90700 1 -- 172.21.15.58:0/2573318356 shutdown_connections 2021-11-20T05:38:52.075 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.076+0000 7f71c1a90700 1 -- 172.21.15.58:0/2573318356 wait complete. 2021-11-20T05:38:52.075 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.077+0000 7f71c1a90700 1 Processor -- start 2021-11-20T05:38:52.076 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.077+0000 7f71c1a90700 1 -- start start 2021-11-20T05:38:52.076 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.077+0000 7f71c1a90700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71bc0f6680 0x7f71bc106e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:52.076 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.077+0000 7f71c1a90700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f71bc0f7920 0x7f71bc1072f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:52.076 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.077+0000 7f71c1a90700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc1078a0 con 0x7f71bc0f7920 2021-11-20T05:38:52.077 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.077+0000 7f71c1a90700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc1079e0 con 0x7f71bc0f6680 2021-11-20T05:38:52.077 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.077+0000 7f71baffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f71bc0f7920 0x7f71bc1072f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:52.077 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.077+0000 7f71baffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f71bc0f7920 0x7f71bc1072f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42394/0 (socket says 172.21.15.58:42394) 2021-11-20T05:38:52.077 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.077+0000 7f71baffd700 1 -- 172.21.15.58:0/4261815679 learned_addr learned my addr 172.21.15.58:0/4261815679 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:52.077 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.078+0000 7f71bb7fe700 1 --2- 172.21.15.58:0/4261815679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71bc0f6680 0x7f71bc106e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:52.078 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.078+0000 7f71baffd700 1 -- 172.21.15.58:0/4261815679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71bc0f6680 msgr2=0x7f71bc106e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:52.078 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.078+0000 7f71baffd700 1 --2- 172.21.15.58:0/4261815679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71bc0f6680 0x7f71bc106e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:52.078 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.078+0000 7f71baffd700 1 -- 172.21.15.58:0/4261815679 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b0005040 con 0x7f71bc0f7920 2021-11-20T05:38:52.078 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.078+0000 7f71baffd700 1 --2- 172.21.15.58:0/4261815679 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f71bc0f7920 0x7f71bc1072f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f71b0009990 tx=0x7f71b0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:52.079 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.078+0000 7f71a3fff700 1 -- 172.21.15.58:0/4261815679 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71b0018070 con 0x7f71bc0f7920 2021-11-20T05:38:52.079 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.078+0000 7f71a3fff700 1 -- 172.21.15.58:0/4261815679 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f71b0013410 con 0x7f71bc0f7920 2021-11-20T05:38:52.079 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.078+0000 7f71c1a90700 1 -- 172.21.15.58:0/4261815679 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71bc10c420 con 0x7f71bc0f7920 2021-11-20T05:38:52.079 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.078+0000 7f71a3fff700 1 -- 172.21.15.58:0/4261815679 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71b001c3c0 con 0x7f71bc0f7920 2021-11-20T05:38:52.079 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.078+0000 7f71c1a90700 1 -- 172.21.15.58:0/4261815679 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71bc10c9e0 con 0x7f71bc0f7920 2021-11-20T05:38:52.081 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.079+0000 7f71c1a90700 1 -- 172.21.15.58:0/4261815679 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71bc027a00 con 0x7f71bc0f7920 2021-11-20T05:38:52.081 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.080+0000 7f71a3fff700 1 -- 172.21.15.58:0/4261815679 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f71b001d420 con 0x7f71bc0f7920 2021-11-20T05:38:52.081 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.080+0000 7f71a3fff700 1 --2- 172.21.15.58:0/4261815679 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f71a405c2a0 0x7f71a405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:52.082 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.080+0000 7f71a3fff700 1 -- 172.21.15.58:0/4261815679 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f71b0078150 con 0x7f71bc0f7920 2021-11-20T05:38:52.082 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.080+0000 7f71bb7fe700 1 --2- 172.21.15.58:0/4261815679 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f71a405c2a0 0x7f71a405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:52.082 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.081+0000 7f71bb7fe700 1 --2- 172.21.15.58:0/4261815679 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f71a405c2a0 0x7f71a405e760 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f71bc0f7780 tx=0x7f71ac00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:52.082 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.084+0000 7f71a3fff700 1 -- 172.21.15.58:0/4261815679 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71b004a850 con 0x7f71bc0f7920 2021-11-20T05:38:52.293 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.298+0000 7f71c1a90700 1 -- 172.21.15.58:0/4261815679 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71bc10c5b0 con 0x7f71bc0f7920 2021-11-20T05:38:52.294 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.299+0000 7f71a3fff700 1 -- 172.21.15.58:0/4261815679 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2200 (secure 0 0 0) 0x7f71bc10c5b0 con 0x7f71bc0f7920 2021-11-20T05:38:52.295 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:52.295 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:52.287814+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:52.296 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.300+0000 7f71c1a90700 1 -- 172.21.15.58:0/4261815679 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f71a405c2a0 msgr2=0x7f71a405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:52.298 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.300+0000 7f71c1a90700 1 --2- 172.21.15.58:0/4261815679 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f71a405c2a0 0x7f71a405e760 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f71bc0f7780 tx=0x7f71ac00b040).stop 2021-11-20T05:38:52.298 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.300+0000 7f71c1a90700 1 -- 172.21.15.58:0/4261815679 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f71bc0f7920 msgr2=0x7f71bc1072f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:52.299 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.300+0000 7f71c1a90700 1 --2- 172.21.15.58:0/4261815679 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f71bc0f7920 0x7f71bc1072f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f71b0009990 tx=0x7f71b0009b30).stop 2021-11-20T05:38:52.299 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.301+0000 7f71c1a90700 1 -- 172.21.15.58:0/4261815679 shutdown_connections 2021-11-20T05:38:52.299 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.301+0000 7f71c1a90700 1 --2- 172.21.15.58:0/4261815679 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f71a405c2a0 0x7f71a405e760 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:52.299 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.301+0000 7f71c1a90700 1 --2- 172.21.15.58:0/4261815679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71bc0f6680 0x7f71bc106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:52.300 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.301+0000 7f71c1a90700 1 --2- 172.21.15.58:0/4261815679 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f71bc0f7920 0x7f71bc1072f0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:52.300 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.301+0000 7f71c1a90700 1 -- 172.21.15.58:0/4261815679 >> 172.21.15.58:0/4261815679 conn(0x7f71bc0f1de0 msgr2=0x7f71bc0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:52.300 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.301+0000 7f71c1a90700 1 -- 172.21.15.58:0/4261815679 shutdown_connections 2021-11-20T05:38:52.300 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:52.301+0000 7f71c1a90700 1 -- 172.21.15.58:0/4261815679 wait complete. 2021-11-20T05:38:52.301 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 100 2021-11-20T05:38:52.309 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:52.287814+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:52.309 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:rejoin 2021-11-20T05:38:52.661 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:52 smithi187 conmon[36358]: cluster 2021-11-20T05:38:51.309809+0000 mds.cephfs.smithi187.wxduhu (mds.0) 1 : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0219997 2021-11-20T05:38:52.662 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:52 smithi187 conmon[36358]: cluster 2021-11-20T05:38:52.291005+0000 mon.smithi058 (mon.0) 887 : cluster [DBG] mds.? [v2:172.21.15.187:6824/1414418853,v1:172.21.15.187:6825/1414418853] up:rejoin 2021-11-20T05:38:52.662 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:52 smithi187 conmon[36358]: cluster 2021-11-20T05:38:52.291075+0000 mon.smithi058 (mon.0) 888 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2021-11-20T05:38:52.662 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:52 smithi187 conmon[36358]: audit 2021-11-20T05:38:52.299647+0000 mon.smithi058 (mon.0) 889 : audit [DBG] from='client.? 172.21.15.58:0/4261815679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:52.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:52 smithi058 conmon[33537]: cluster 2021-11-20T05:38:51.309809+0000 mds.cephfs.smithi187.wxduhu ( 2021-11-20T05:38:52.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:52 smithi058 conmon[33537]: mds.0) 1 : cluster [DBG] reconnect by client.24337 v1:172.21.15.58:0/537856901 after 0.0219997 2021-11-20T05:38:52.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:52 smithi058 conmon[33537]: cluster 2021-11-20T05:38:52.291005+0000 mon.smithi058 (mon.0) 887 : cluster [DBG] mds.? [v2:172.21.15.187:6824/1414418853,v1:172.21.15.187:6825/1414418853] up:rejoin 2021-11-20T05:38:52.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:52 smithi058 conmon[33537]: cluster 2021-11-20T05:38:52.291075+0000 mon.smithi058 (mon.0) 888 2021-11-20T05:38:52.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:52 smithi058 conmon[33537]: : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin,1=cephfs.smithi058.nuxpgf=up:active} 2021-11-20T05:38:52.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:52 smithi058 conmon[33537]: audit 2021-11-20T05:38:52.299647+0000 mon.smithi058 (mon.0) 889 : audit 2021-11-20T05:38:52.717 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:52 smithi058 conmon[33537]: [DBG] from='client.? 172.21.15.58:0/4261815679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:53.311 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:53.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.449+0000 7f3a9a431700 1 -- 172.21.15.58:0/3614266698 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3a940f5580 msgr2=0x7f3a940f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:53.447 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.449+0000 7f3a9a431700 1 --2- 172.21.15.58:0/3614266698 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3a940f5580 0x7f3a940f59a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f3a84004660 tx=0x7f3a84009b30).stop 2021-11-20T05:38:53.448 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.449+0000 7f3a9a431700 1 -- 172.21.15.58:0/3614266698 shutdown_connections 2021-11-20T05:38:53.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.449+0000 7f3a9a431700 1 --2- 172.21.15.58:0/3614266698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a940f6820 0x7f3a940f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:53.449 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.449+0000 7f3a9a431700 1 --2- 172.21.15.58:0/3614266698 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3a940f5580 0x7f3a940f59a0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:53.450 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.449+0000 7f3a9a431700 1 -- 172.21.15.58:0/3614266698 >> 172.21.15.58:0/3614266698 conn(0x7f3a940f0ce0 msgr2=0x7f3a940f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:53.450 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.449+0000 7f3a9a431700 1 -- 172.21.15.58:0/3614266698 shutdown_connections 2021-11-20T05:38:53.450 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.449+0000 7f3a9a431700 1 -- 172.21.15.58:0/3614266698 wait complete. 2021-11-20T05:38:53.450 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.450+0000 7f3a9a431700 1 Processor -- start 2021-11-20T05:38:53.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.450+0000 7f3a9a431700 1 -- start start 2021-11-20T05:38:53.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.450+0000 7f3a9a431700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3a940f5580 0x7f3a94060550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:53.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.450+0000 7f3a9a431700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a940f6820 0x7f3a9410e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:53.451 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.450+0000 7f3a9a431700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a94107a40 con 0x7f3a940f5580 2021-11-20T05:38:53.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.450+0000 7f3a9a431700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9410e550 con 0x7f3a940f6820 2021-11-20T05:38:53.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.450+0000 7f3a93fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3a940f5580 0x7f3a94060550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:53.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.450+0000 7f3a93fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3a940f5580 0x7f3a94060550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42410/0 (socket says 172.21.15.58:42410) 2021-11-20T05:38:53.452 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.450+0000 7f3a93fff700 1 -- 172.21.15.58:0/2144345731 learned_addr learned my addr 172.21.15.58:0/2144345731 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:53.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.451+0000 7f3a93fff700 1 -- 172.21.15.58:0/2144345731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a940f6820 msgr2=0x7f3a9410e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:53.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.451+0000 7f3a937fe700 1 --2- 172.21.15.58:0/2144345731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a940f6820 0x7f3a9410e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:53.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.451+0000 7f3a93fff700 1 --2- 172.21.15.58:0/2144345731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a940f6820 0x7f3a9410e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:53.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.451+0000 7f3a93fff700 1 -- 172.21.15.58:0/2144345731 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a84005040 con 0x7f3a940f5580 2021-11-20T05:38:53.453 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.451+0000 7f3a93fff700 1 --2- 172.21.15.58:0/2144345731 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3a940f5580 0x7f3a94060550 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f3a84009f50 tx=0x7f3a84009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:53.454 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.451+0000 7f3a90ff9700 1 -- 172.21.15.58:0/2144345731 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a84018070 con 0x7f3a940f5580 2021-11-20T05:38:53.454 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.451+0000 7f3a9a431700 1 -- 172.21.15.58:0/2144345731 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a9410e6f0 con 0x7f3a940f5580 2021-11-20T05:38:53.454 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.451+0000 7f3a90ff9700 1 -- 172.21.15.58:0/2144345731 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a84013410 con 0x7f3a940f5580 2021-11-20T05:38:53.454 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.451+0000 7f3a90ff9700 1 -- 172.21.15.58:0/2144345731 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a8401c620 con 0x7f3a940f5580 2021-11-20T05:38:53.455 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.451+0000 7f3a9a431700 1 -- 172.21.15.58:0/2144345731 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a9410ec20 con 0x7f3a940f5580 2021-11-20T05:38:53.456 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.452+0000 7f3a9a431700 1 -- 172.21.15.58:0/2144345731 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a94046330 con 0x7f3a940f5580 2021-11-20T05:38:53.456 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.452+0000 7f3a90ff9700 1 -- 172.21.15.58:0/2144345731 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3a8401c890 con 0x7f3a940f5580 2021-11-20T05:38:53.456 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.453+0000 7f3a90ff9700 1 --2- 172.21.15.58:0/2144345731 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3a7c05c2a0 0x7f3a7c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:53.457 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.453+0000 7f3a90ff9700 1 -- 172.21.15.58:0/2144345731 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f3a840782f0 con 0x7f3a940f5580 2021-11-20T05:38:53.457 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.453+0000 7f3a937fe700 1 --2- 172.21.15.58:0/2144345731 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3a7c05c2a0 0x7f3a7c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:53.457 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.453+0000 7f3a937fe700 1 --2- 172.21.15.58:0/2144345731 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3a7c05c2a0 0x7f3a7c05e760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f3a880041d0 tx=0x7f3a88009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:53.457 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.457+0000 7f3a90ff9700 1 -- 172.21.15.58:0/2144345731 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a8404aa50 con 0x7f3a940f5580 2021-11-20T05:38:53.530 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:53 smithi058 conmon[33537]: cluster 2021-11-20T05:38:52.306531+0000 mgr.smithi058.zagdlk (mgr.14164) 335 : cluster [DBG] pgmap v313: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T05:38:53.657 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.661+0000 7f3a9a431700 1 -- 172.21.15.58:0/2144345731 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a940fdd10 con 0x7f3a940f5580 2021-11-20T05:38:53.657 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.661+0000 7f3a90ff9700 1 -- 172.21.15.58:0/2144345731 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2200 (secure 0 0 0) 0x7f3a8404e070 con 0x7f3a940f5580 2021-11-20T05:38:53.658 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:53.658 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:52.287814+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:53.660 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.663+0000 7f3a9a431700 1 -- 172.21.15.58:0/2144345731 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3a7c05c2a0 msgr2=0x7f3a7c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:53.660 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.663+0000 7f3a9a431700 1 --2- 172.21.15.58:0/2144345731 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3a7c05c2a0 0x7f3a7c05e760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f3a880041d0 tx=0x7f3a88009040).stop 2021-11-20T05:38:53.661 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.663+0000 7f3a9a431700 1 -- 172.21.15.58:0/2144345731 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3a940f5580 msgr2=0x7f3a94060550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:53.661 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.663+0000 7f3a9a431700 1 --2- 172.21.15.58:0/2144345731 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3a940f5580 0x7f3a94060550 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f3a84009f50 tx=0x7f3a84009b30).stop 2021-11-20T05:38:53.662 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.663+0000 7f3a9a431700 1 -- 172.21.15.58:0/2144345731 shutdown_connections 2021-11-20T05:38:53.662 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.663+0000 7f3a9a431700 1 --2- 172.21.15.58:0/2144345731 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3a7c05c2a0 0x7f3a7c05e760 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:53.662 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.663+0000 7f3a9a431700 1 --2- 172.21.15.58:0/2144345731 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3a940f5580 0x7f3a94060550 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:53.662 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.663+0000 7f3a9a431700 1 --2- 172.21.15.58:0/2144345731 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a940f6820 0x7f3a9410e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:53.663 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.663+0000 7f3a9a431700 1 -- 172.21.15.58:0/2144345731 >> 172.21.15.58:0/2144345731 conn(0x7f3a940f0ce0 msgr2=0x7f3a940fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:53.663 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.663+0000 7f3a9a431700 1 -- 172.21.15.58:0/2144345731 shutdown_connections 2021-11-20T05:38:53.663 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:53.663+0000 7f3a9a431700 1 -- 172.21.15.58:0/2144345731 wait complete. 2021-11-20T05:38:53.663 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 100 2021-11-20T05:38:53.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:52.287814+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:53.671 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:rejoin 2021-11-20T05:38:53.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:53 smithi187 conmon[36358]: cluster 2021-11-20T05:38:52.306531+0000 mgr.smithi058.zagdlk (mgr.14164) 335 : cluster [DBG] pgmap v313: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T05:38:54.673 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:54.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:54 smithi187 conmon[36358]: audit 2021-11-20T05:38:53.662227+0000 mon.smithi058 (mon.0) 890 : audit [DBG] from='client.? 172.21.15.58:0/2144345731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:54.702 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:54 smithi058 conmon[33537]: audit 2021-11-20T05:38:53.662227+0000 mon.smithi058 (mon.0) 890 : audit 2021-11-20T05:38:54.702 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:54 smithi058 conmon[33537]: [DBG] from='client.? 172.21.15.58:0/2144345731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:54.808 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.811+0000 7fa894576700 1 -- 172.21.15.58:0/4025652584 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa88c0f6680 msgr2=0x7fa88c0f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:54.809 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.811+0000 7fa894576700 1 --2- 172.21.15.58:0/4025652584 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa88c0f6680 0x7fa88c0f6aa0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7fa880004660 tx=0x7fa880009b30).stop 2021-11-20T05:38:54.809 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.811+0000 7fa894576700 1 -- 172.21.15.58:0/4025652584 shutdown_connections 2021-11-20T05:38:54.809 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.811+0000 7fa894576700 1 --2- 172.21.15.58:0/4025652584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa88c0f7920 0x7fa88c0f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:54.809 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.811+0000 7fa894576700 1 --2- 172.21.15.58:0/4025652584 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa88c0f6680 0x7fa88c0f6aa0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:54.810 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.811+0000 7fa894576700 1 -- 172.21.15.58:0/4025652584 >> 172.21.15.58:0/4025652584 conn(0x7fa88c0f1de0 msgr2=0x7fa88c0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:54.810 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.811+0000 7fa894576700 1 -- 172.21.15.58:0/4025652584 shutdown_connections 2021-11-20T05:38:54.810 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.811+0000 7fa894576700 1 -- 172.21.15.58:0/4025652584 wait complete. 2021-11-20T05:38:54.810 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.812+0000 7fa894576700 1 Processor -- start 2021-11-20T05:38:54.811 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.812+0000 7fa894576700 1 -- start start 2021-11-20T05:38:54.811 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.812+0000 7fa894576700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa88c0f6680 0x7fa88c106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:54.811 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.812+0000 7fa894576700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa88c0f7920 0x7fa88c1073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:54.812 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.812+0000 7fa894576700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c107950 con 0x7fa88c0f6680 2021-11-20T05:38:54.812 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.812+0000 7fa894576700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c107a90 con 0x7fa88c0f7920 2021-11-20T05:38:54.812 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.812+0000 7fa892312700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa88c0f6680 0x7fa88c106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:54.812 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.812+0000 7fa892312700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa88c0f6680 0x7fa88c106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42426/0 (socket says 172.21.15.58:42426) 2021-11-20T05:38:54.812 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.812+0000 7fa892312700 1 -- 172.21.15.58:0/1575706441 learned_addr learned my addr 172.21.15.58:0/1575706441 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:54.813 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.812+0000 7fa891b11700 1 --2- 172.21.15.58:0/1575706441 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa88c0f7920 0x7fa88c1073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:54.813 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.812+0000 7fa892312700 1 -- 172.21.15.58:0/1575706441 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa88c0f7920 msgr2=0x7fa88c1073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:54.813 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.812+0000 7fa892312700 1 --2- 172.21.15.58:0/1575706441 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa88c0f7920 0x7fa88c1073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:54.813 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.812+0000 7fa892312700 1 -- 172.21.15.58:0/1575706441 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa880005040 con 0x7fa88c0f6680 2021-11-20T05:38:54.814 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.813+0000 7fa892312700 1 --2- 172.21.15.58:0/1575706441 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa88c0f6680 0x7fa88c106eb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7fa880007a60 tx=0x7fa880009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:54.814 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.813+0000 7fa87effd700 1 -- 172.21.15.58:0/1575706441 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa880018070 con 0x7fa88c0f6680 2021-11-20T05:38:54.814 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.813+0000 7fa87effd700 1 -- 172.21.15.58:0/1575706441 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa880013410 con 0x7fa88c0f6680 2021-11-20T05:38:54.814 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.813+0000 7fa87effd700 1 -- 172.21.15.58:0/1575706441 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa88001c620 con 0x7fa88c0f6680 2021-11-20T05:38:54.814 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.813+0000 7fa894576700 1 -- 172.21.15.58:0/1575706441 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa88c10c4d0 con 0x7fa88c0f6680 2021-11-20T05:38:54.815 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.813+0000 7fa894576700 1 -- 172.21.15.58:0/1575706441 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa88c10ca90 con 0x7fa88c0f6680 2021-11-20T05:38:54.816 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.814+0000 7fa894576700 1 -- 172.21.15.58:0/1575706441 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa88c027a00 con 0x7fa88c0f6680 2021-11-20T05:38:54.816 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.814+0000 7fa87effd700 1 -- 172.21.15.58:0/1575706441 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa88001d420 con 0x7fa88c0f6680 2021-11-20T05:38:54.816 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.814+0000 7fa87effd700 1 --2- 172.21.15.58:0/1575706441 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa87805c2a0 0x7fa87805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:54.817 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.815+0000 7fa87effd700 1 -- 172.21.15.58:0/1575706441 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fa880078210 con 0x7fa88c0f6680 2021-11-20T05:38:54.817 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.815+0000 7fa891b11700 1 --2- 172.21.15.58:0/1575706441 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa87805c2a0 0x7fa87805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:54.817 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.815+0000 7fa891b11700 1 --2- 172.21.15.58:0/1575706441 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa87805c2a0 0x7fa87805e760 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fa8880096c0 tx=0x7fa888009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:54.817 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:54.818+0000 7fa87effd700 1 -- 172.21.15.58:0/1575706441 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa88004a9f0 con 0x7fa88c0f6680 2021-11-20T05:38:55.009 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:55.014+0000 7fa894576700 1 -- 172.21.15.58:0/1575706441 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa88c0fee10 con 0x7fa88c0f6680 2021-11-20T05:38:55.010 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:55.014+0000 7fa87effd700 1 -- 172.21.15.58:0/1575706441 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2200 (secure 0 0 0) 0x7fa88004e010 con 0x7fa88c0f6680 2021-11-20T05:38:55.010 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:55.010 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:52.287814+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:55.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:55.016+0000 7fa894576700 1 -- 172.21.15.58:0/1575706441 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa87805c2a0 msgr2=0x7fa87805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:55.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:55.016+0000 7fa894576700 1 --2- 172.21.15.58:0/1575706441 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa87805c2a0 0x7fa87805e760 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fa8880096c0 tx=0x7fa888009be0).stop 2021-11-20T05:38:55.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:55.016+0000 7fa894576700 1 -- 172.21.15.58:0/1575706441 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa88c0f6680 msgr2=0x7fa88c106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:55.012 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:55.016+0000 7fa894576700 1 --2- 172.21.15.58:0/1575706441 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa88c0f6680 0x7fa88c106eb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7fa880007a60 tx=0x7fa880009b30).stop 2021-11-20T05:38:55.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:55.016+0000 7fa894576700 1 -- 172.21.15.58:0/1575706441 shutdown_connections 2021-11-20T05:38:55.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:55.016+0000 7fa894576700 1 --2- 172.21.15.58:0/1575706441 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fa87805c2a0 0x7fa87805e760 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:55.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:55.016+0000 7fa894576700 1 --2- 172.21.15.58:0/1575706441 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fa88c0f6680 0x7fa88c106eb0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:55.013 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:55.016+0000 7fa894576700 1 --2- 172.21.15.58:0/1575706441 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa88c0f7920 0x7fa88c1073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:55.014 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:55.016+0000 7fa894576700 1 -- 172.21.15.58:0/1575706441 >> 172.21.15.58:0/1575706441 conn(0x7fa88c0f1de0 msgr2=0x7fa88c0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:55.014 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:55.016+0000 7fa894576700 1 -- 172.21.15.58:0/1575706441 shutdown_connections 2021-11-20T05:38:55.014 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:55.016+0000 7fa894576700 1 -- 172.21.15.58:0/1575706441 wait complete. 2021-11-20T05:38:55.014 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 100 2021-11-20T05:38:55.023 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:52.287814+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:55.024 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:rejoin 2021-11-20T05:38:55.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:55 smithi187 conmon[36358]: cluster 2021-11-20T05:38:54.307106+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:38:55.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:55 smithi187 conmon[36358]: ) 336 : cluster [DBG] pgmap v314: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:55.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:55 smithi187 conmon[36358]: audit 2021-11-20T05:38:55.015522+0000 mon.smithi058 (mon.0) 891 : audit [DBG] from='client.? 172.21.15.58:0/1575706441' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:55.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:55 smithi058 conmon[33537]: cluster 2021-11-20T05:38:54.307106+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:38:55.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:55 smithi058 conmon[33537]: mgr.14164) 336 : cluster [DBG] pgmap v314: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:55.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:55 smithi058 conmon[33537]: audit 2021-11-20T05:38:55.015522+0000 mon.smithi058 (mon.0) 891 : audit [DBG] from='client.? 172.21.15.58:0/1575706441' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:56.025 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:56.161 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.162+0000 7fd1c3f92700 1 -- 172.21.15.58:0/3228690478 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1bc0f6670 msgr2=0x7fd1bc0f6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:56.161 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.162+0000 7fd1c3f92700 1 --2- 172.21.15.58:0/3228690478 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1bc0f6670 0x7fd1bc0f6a90 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7fd1b4004660 tx=0x7fd1b4009b30).stop 2021-11-20T05:38:56.161 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.162+0000 7fd1c3f92700 1 -- 172.21.15.58:0/3228690478 shutdown_connections 2021-11-20T05:38:56.161 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.162+0000 7fd1c3f92700 1 --2- 172.21.15.58:0/3228690478 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1bc0f7910 0x7fd1bc0f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:56.162 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.162+0000 7fd1c3f92700 1 --2- 172.21.15.58:0/3228690478 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1bc0f6670 0x7fd1bc0f6a90 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:56.162 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.162+0000 7fd1c3f92700 1 -- 172.21.15.58:0/3228690478 >> 172.21.15.58:0/3228690478 conn(0x7fd1bc0f1db0 msgr2=0x7fd1bc0f41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:56.162 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.162+0000 7fd1c3f92700 1 -- 172.21.15.58:0/3228690478 shutdown_connections 2021-11-20T05:38:56.162 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.162+0000 7fd1c3f92700 1 -- 172.21.15.58:0/3228690478 wait complete. 2021-11-20T05:38:56.163 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.163+0000 7fd1c3f92700 1 Processor -- start 2021-11-20T05:38:56.163 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.163+0000 7fd1c3f92700 1 -- start start 2021-11-20T05:38:56.163 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.163+0000 7fd1c3f92700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1bc0f6670 0x7fd1bc106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:56.163 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.163+0000 7fd1c3f92700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1bc0f7910 0x7fd1bc1073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:56.164 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.163+0000 7fd1c3f92700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc107960 con 0x7fd1bc0f7910 2021-11-20T05:38:56.164 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.163+0000 7fd1c3f92700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc107aa0 con 0x7fd1bc0f6670 2021-11-20T05:38:56.164 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.163+0000 7fd1c152d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1bc0f7910 0x7fd1bc1073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:56.164 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.163+0000 7fd1c152d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1bc0f7910 0x7fd1bc1073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42442/0 (socket says 172.21.15.58:42442) 2021-11-20T05:38:56.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.163+0000 7fd1c152d700 1 -- 172.21.15.58:0/1305645544 learned_addr learned my addr 172.21.15.58:0/1305645544 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:56.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.163+0000 7fd1c1d2e700 1 --2- 172.21.15.58:0/1305645544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1bc0f6670 0x7fd1bc106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:56.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.164+0000 7fd1c152d700 1 -- 172.21.15.58:0/1305645544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1bc0f6670 msgr2=0x7fd1bc106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:56.165 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.164+0000 7fd1c152d700 1 --2- 172.21.15.58:0/1305645544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1bc0f6670 0x7fd1bc106ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:56.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.164+0000 7fd1c152d700 1 -- 172.21.15.58:0/1305645544 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b4005040 con 0x7fd1bc0f7910 2021-11-20T05:38:56.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.164+0000 7fd1c152d700 1 --2- 172.21.15.58:0/1305645544 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1bc0f7910 0x7fd1bc1073b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7fd1b00026d0 tx=0x7fd1b0009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:56.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.164+0000 7fd1ba7fc700 1 -- 172.21.15.58:0/1305645544 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1b00079c0 con 0x7fd1bc0f7910 2021-11-20T05:38:56.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.164+0000 7fd1ba7fc700 1 -- 172.21.15.58:0/1305645544 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1b000f040 con 0x7fd1bc0f7910 2021-11-20T05:38:56.166 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.164+0000 7fd1c3f92700 1 -- 172.21.15.58:0/1305645544 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1bc10c540 con 0x7fd1bc0f7910 2021-11-20T05:38:56.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.164+0000 7fd1ba7fc700 1 -- 172.21.15.58:0/1305645544 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1b00148b0 con 0x7fd1bc0f7910 2021-11-20T05:38:56.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.164+0000 7fd1c3f92700 1 -- 172.21.15.58:0/1305645544 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc10cb00 con 0x7fd1bc0f7910 2021-11-20T05:38:56.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.165+0000 7fd1c3f92700 1 -- 172.21.15.58:0/1305645544 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1bc027a00 con 0x7fd1bc0f7910 2021-11-20T05:38:56.167 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.165+0000 7fd1ba7fc700 1 -- 172.21.15.58:0/1305645544 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd1b0007b20 con 0x7fd1bc0f7910 2021-11-20T05:38:56.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.166+0000 7fd1ba7fc700 1 --2- 172.21.15.58:0/1305645544 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd1a405c2a0 0x7fd1a405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:56.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.166+0000 7fd1ba7fc700 1 -- 172.21.15.58:0/1305645544 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fd1b0076340 con 0x7fd1bc0f7910 2021-11-20T05:38:56.168 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.166+0000 7fd1c1d2e700 1 --2- 172.21.15.58:0/1305645544 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd1a405c2a0 0x7fd1a405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:56.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.166+0000 7fd1c1d2e700 1 --2- 172.21.15.58:0/1305645544 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd1a405c2a0 0x7fd1a405e760 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fd1b4009f50 tx=0x7fd1b4009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:56.169 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.170+0000 7fd1ba7fc700 1 -- 172.21.15.58:0/1305645544 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd1b0048b20 con 0x7fd1bc0f7910 2021-11-20T05:38:56.359 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.364+0000 7fd1c3f92700 1 -- 172.21.15.58:0/1305645544 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1bc0fee00 con 0x7fd1bc0f7910 2021-11-20T05:38:56.360 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.364+0000 7fd1ba7fc700 1 -- 172.21.15.58:0/1305645544 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2200 (secure 0 0 0) 0x7fd1b004c140 con 0x7fd1bc0f7910 2021-11-20T05:38:56.360 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:56.360 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:52.287814+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:56.363 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.366+0000 7fd1c3f92700 1 -- 172.21.15.58:0/1305645544 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd1a405c2a0 msgr2=0x7fd1a405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:56.363 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.366+0000 7fd1c3f92700 1 --2- 172.21.15.58:0/1305645544 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd1a405c2a0 0x7fd1a405e760 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fd1b4009f50 tx=0x7fd1b4009b30).stop 2021-11-20T05:38:56.363 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.366+0000 7fd1c3f92700 1 -- 172.21.15.58:0/1305645544 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1bc0f7910 msgr2=0x7fd1bc1073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:56.364 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.366+0000 7fd1c3f92700 1 --2- 172.21.15.58:0/1305645544 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1bc0f7910 0x7fd1bc1073b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7fd1b00026d0 tx=0x7fd1b0009de0).stop 2021-11-20T05:38:56.364 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.366+0000 7fd1c3f92700 1 -- 172.21.15.58:0/1305645544 shutdown_connections 2021-11-20T05:38:56.364 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.366+0000 7fd1c3f92700 1 --2- 172.21.15.58:0/1305645544 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd1a405c2a0 0x7fd1a405e760 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:56.364 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.366+0000 7fd1c3f92700 1 --2- 172.21.15.58:0/1305645544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1bc0f6670 0x7fd1bc106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:56.364 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.366+0000 7fd1c3f92700 1 --2- 172.21.15.58:0/1305645544 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1bc0f7910 0x7fd1bc1073b0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:56.365 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.366+0000 7fd1c3f92700 1 -- 172.21.15.58:0/1305645544 >> 172.21.15.58:0/1305645544 conn(0x7fd1bc0f1db0 msgr2=0x7fd1bc0fb360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:56.365 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.366+0000 7fd1c3f92700 1 -- 172.21.15.58:0/1305645544 shutdown_connections 2021-11-20T05:38:56.365 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:56.366+0000 7fd1c3f92700 1 -- 172.21.15.58:0/1305645544 wait complete. 2021-11-20T05:38:56.366 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 100 2021-11-20T05:38:56.373 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:52.287814+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:56.373 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:rejoin 2021-11-20T05:38:56.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:56 smithi187 conmon[36358]: audit 2021-11-20T05:38:56.365435+0000 mon.smithi058 (mon.0) 892 : audit [DBG] from='client.? 172.21.15.58:0/1305645544' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:56.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:56 smithi187 conmon[36358]: 2021-11-20T05:38:56.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:56 smithi058 conmon[33537]: audit 2021-11-20T05:38:56.365435+0000 mon.smithi058 (mon.0 2021-11-20T05:38:56.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:56 smithi058 conmon[33537]: ) 892 : audit [DBG] from='client.? 172.21.15.58:0/1305645544' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:57.375 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:57.465 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:57 smithi058 conmon[33537]: cluster 2021-11-20T05:38:56.307570+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:38:57.465 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:57 smithi058 conmon[33537]: mgr.14164) 337 : cluster [DBG] pgmap v315: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:57.510 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.513+0000 7f14c0939700 1 -- 172.21.15.58:0/367464056 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14bc0f7870 msgr2=0x7f14bc0f7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:57.510 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.513+0000 7f14c0939700 1 --2- 172.21.15.58:0/367464056 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14bc0f7870 0x7f14bc0f7cd0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f14b0004660 tx=0x7f14b0009b30).stop 2021-11-20T05:38:57.510 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.513+0000 7f14c0939700 1 -- 172.21.15.58:0/367464056 shutdown_connections 2021-11-20T05:38:57.511 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.513+0000 7f14c0939700 1 --2- 172.21.15.58:0/367464056 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14bc0f7870 0x7f14bc0f7cd0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:57.511 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.513+0000 7f14c0939700 1 --2- 172.21.15.58:0/367464056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14bc0f65d0 0x7f14bc0f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:57.511 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.513+0000 7f14c0939700 1 -- 172.21.15.58:0/367464056 >> 172.21.15.58:0/367464056 conn(0x7f14bc0f1d10 msgr2=0x7f14bc0f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:57.512 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.513+0000 7f14c0939700 1 -- 172.21.15.58:0/367464056 shutdown_connections 2021-11-20T05:38:57.512 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.513+0000 7f14c0939700 1 -- 172.21.15.58:0/367464056 wait complete. 2021-11-20T05:38:57.512 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.513+0000 7f14c0939700 1 Processor -- start 2021-11-20T05:38:57.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.513+0000 7f14c0939700 1 -- start start 2021-11-20T05:38:57.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.514+0000 7f14c0939700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14bc0f65d0 0x7f14bc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:57.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.514+0000 7f14c0939700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14bc0f7870 0x7f14bc069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:57.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.514+0000 7f14c0939700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc06a380 con 0x7f14bc0f7870 2021-11-20T05:38:57.513 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.514+0000 7f14c0939700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc06a4c0 con 0x7f14bc0f65d0 2021-11-20T05:38:57.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.514+0000 7f14baffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14bc0f7870 0x7f14bc069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:57.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.514+0000 7f14baffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14bc0f7870 0x7f14bc069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42454/0 (socket says 172.21.15.58:42454) 2021-11-20T05:38:57.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.514+0000 7f14baffd700 1 -- 172.21.15.58:0/3794146668 learned_addr learned my addr 172.21.15.58:0/3794146668 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:57.514 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.514+0000 7f14bb7fe700 1 --2- 172.21.15.58:0/3794146668 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14bc0f65d0 0x7f14bc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:57.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.514+0000 7f14baffd700 1 -- 172.21.15.58:0/3794146668 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14bc0f65d0 msgr2=0x7f14bc06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:57.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.514+0000 7f14baffd700 1 --2- 172.21.15.58:0/3794146668 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14bc0f65d0 0x7f14bc06b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:57.515 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.514+0000 7f14baffd700 1 -- 172.21.15.58:0/3794146668 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b0005040 con 0x7f14bc0f7870 2021-11-20T05:38:57.516 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.514+0000 7f14baffd700 1 --2- 172.21.15.58:0/3794146668 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14bc0f7870 0x7f14bc069e20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f14b0009830 tx=0x7f14b0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:57.516 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.514+0000 7f14a3fff700 1 -- 172.21.15.58:0/3794146668 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14b0018070 con 0x7f14bc0f7870 2021-11-20T05:38:57.516 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.514+0000 7f14a3fff700 1 -- 172.21.15.58:0/3794146668 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14b0013410 con 0x7f14bc0f7870 2021-11-20T05:38:57.516 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.514+0000 7f14c0939700 1 -- 172.21.15.58:0/3794146668 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14bc06a740 con 0x7f14bc0f7870 2021-11-20T05:38:57.517 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.514+0000 7f14a3fff700 1 -- 172.21.15.58:0/3794146668 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14b001c620 con 0x7f14bc0f7870 2021-11-20T05:38:57.517 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.515+0000 7f14c0939700 1 -- 172.21.15.58:0/3794146668 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14bc06aca0 con 0x7f14bc0f7870 2021-11-20T05:38:57.518 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.515+0000 7f14c0939700 1 -- 172.21.15.58:0/3794146668 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14bc045ad0 con 0x7f14bc0f7870 2021-11-20T05:38:57.519 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.516+0000 7f14a3fff700 1 -- 172.21.15.58:0/3794146668 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f14b001c8e0 con 0x7f14bc0f7870 2021-11-20T05:38:57.519 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.516+0000 7f14a3fff700 1 --2- 172.21.15.58:0/3794146668 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f14a405c350 0x7f14a405e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:57.519 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.516+0000 7f14a3fff700 1 -- 172.21.15.58:0/3794146668 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f14b00783c0 con 0x7f14bc0f7870 2021-11-20T05:38:57.520 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.516+0000 7f14bb7fe700 1 --2- 172.21.15.58:0/3794146668 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f14a405c350 0x7f14a405e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:57.521 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.517+0000 7f14bb7fe700 1 --2- 172.21.15.58:0/3794146668 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f14a405c350 0x7f14a405e810 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f14bc06b730 tx=0x7f14ac009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:57.521 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.520+0000 7f14a3fff700 1 -- 172.21.15.58:0/3794146668 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14b004aa00 con 0x7f14bc0f7870 2021-11-20T05:38:57.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:57 smithi187 conmon[36358]: cluster 2021-11-20T05:38:56.307570+0000 mgr.smithi058.zagdlk (mgr.14164) 337 : cluster [DBG] pgmap v315: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:57.709 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.714+0000 7f14c0939700 1 -- 172.21.15.58:0/3794146668 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14bc027a00 con 0x7f14bc0f7870 2021-11-20T05:38:57.710 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.715+0000 7f14a3fff700 1 -- 172.21.15.58:0/3794146668 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2200 (secure 0 0 0) 0x7f14b004e020 con 0x7f14bc0f7870 2021-11-20T05:38:57.710 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:57.711 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:52.287814+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:57.712 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.716+0000 7f14c0939700 1 -- 172.21.15.58:0/3794146668 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f14a405c350 msgr2=0x7f14a405e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:57.712 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.716+0000 7f14c0939700 1 --2- 172.21.15.58:0/3794146668 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f14a405c350 0x7f14a405e810 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f14bc06b730 tx=0x7f14ac009040).stop 2021-11-20T05:38:57.713 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.716+0000 7f14c0939700 1 -- 172.21.15.58:0/3794146668 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14bc0f7870 msgr2=0x7f14bc069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:57.713 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.716+0000 7f14c0939700 1 --2- 172.21.15.58:0/3794146668 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14bc0f7870 0x7f14bc069e20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f14b0009830 tx=0x7f14b0009b30).stop 2021-11-20T05:38:57.713 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.716+0000 7f14c0939700 1 -- 172.21.15.58:0/3794146668 shutdown_connections 2021-11-20T05:38:57.713 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.716+0000 7f14c0939700 1 --2- 172.21.15.58:0/3794146668 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f14a405c350 0x7f14a405e810 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:57.713 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.716+0000 7f14c0939700 1 --2- 172.21.15.58:0/3794146668 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14bc0f65d0 0x7f14bc06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:57.714 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.716+0000 7f14c0939700 1 --2- 172.21.15.58:0/3794146668 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f14bc0f7870 0x7f14bc069e20 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:57.714 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.716+0000 7f14c0939700 1 -- 172.21.15.58:0/3794146668 >> 172.21.15.58:0/3794146668 conn(0x7f14bc0f1d10 msgr2=0x7f14bc0fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:57.714 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.717+0000 7f14c0939700 1 -- 172.21.15.58:0/3794146668 shutdown_connections 2021-11-20T05:38:57.714 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:57.717+0000 7f14c0939700 1 -- 172.21.15.58:0/3794146668 wait complete. 2021-11-20T05:38:57.715 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 100 2021-11-20T05:38:57.724 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:52.287814+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:57.724 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:rejoin 2021-11-20T05:38:58.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:58 smithi187 conmon[36358]: audit 2021-11-20T05:38:57.715796+0000 mon.smithi058 (mon.0) 893 : audit [DBG] from='client.? 172.21.15.58:0/3794146668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:58.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:58 smithi058 conmon[33537]: audit 2021-11-20T05:38:57.715796+0000 mon.smithi058 (mon.0) 893 : audit [DBG] from='client.? 172.21.15.58:0/3794146668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:58.726 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:38:58.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.862+0000 7fd10d160700 1 -- 172.21.15.58:0/2014661918 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1080f5580 msgr2=0x7fd1080f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:58.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.862+0000 7fd10d160700 1 --2- 172.21.15.58:0/2014661918 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1080f5580 0x7fd1080f59a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7fd0f8004660 tx=0x7fd0f8009b30).stop 2021-11-20T05:38:58.860 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.862+0000 7fd10d160700 1 -- 172.21.15.58:0/2014661918 shutdown_connections 2021-11-20T05:38:58.861 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.862+0000 7fd10d160700 1 --2- 172.21.15.58:0/2014661918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1080f6820 0x7fd1080f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T05:38:58.861 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.862+0000 7fd10d160700 1 --2- 172.21.15.58:0/2014661918 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1080f5580 0x7fd1080f59a0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:58.861 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.862+0000 7fd10d160700 1 -- 172.21.15.58:0/2014661918 >> 172.21.15.58:0/2014661918 conn(0x7fd1080f0ce0 msgr2=0x7fd1080f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:58.862 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.862+0000 7fd10d160700 1 -- 172.21.15.58:0/2014661918 shutdown_connections 2021-11-20T05:38:58.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.863+0000 7fd10d160700 1 -- 172.21.15.58:0/2014661918 wait complete. 2021-11-20T05:38:58.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.863+0000 7fd10d160700 1 Processor -- start 2021-11-20T05:38:58.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.863+0000 7fd10d160700 1 -- start start 2021-11-20T05:38:58.863 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.863+0000 7fd10d160700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1080f5580 0x7fd108060530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:58.864 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.863+0000 7fd10d160700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1080f6820 0x7fd10810bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:58.864 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.863+0000 7fd10d160700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10810c1d0 con 0x7fd1080f5580 2021-11-20T05:38:58.864 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.863+0000 7fd10d160700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10810c310 con 0x7fd1080f6820 2021-11-20T05:38:58.864 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.863+0000 7fd106d9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1080f5580 0x7fd108060530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:58.865 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.864+0000 7fd106d9d700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1080f5580 0x7fd108060530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42464/0 (socket says 172.21.15.58:42464) 2021-11-20T05:38:58.865 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.864+0000 7fd106d9d700 1 -- 172.21.15.58:0/2905105300 learned_addr learned my addr 172.21.15.58:0/2905105300 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:38:58.865 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.864+0000 7fd10659c700 1 --2- 172.21.15.58:0/2905105300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1080f6820 0x7fd10810bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:58.865 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.864+0000 7fd106d9d700 1 -- 172.21.15.58:0/2905105300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1080f6820 msgr2=0x7fd10810bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:58.866 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.864+0000 7fd106d9d700 1 --2- 172.21.15.58:0/2905105300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1080f6820 0x7fd10810bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:58.866 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.864+0000 7fd106d9d700 1 -- 172.21.15.58:0/2905105300 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f8005040 con 0x7fd1080f5580 2021-11-20T05:38:58.866 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.864+0000 7fd106d9d700 1 --2- 172.21.15.58:0/2905105300 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1080f5580 0x7fd108060530 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7fd0f8009f50 tx=0x7fd0f8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:58.866 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.864+0000 7fd0ef7fe700 1 -- 172.21.15.58:0/2905105300 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0f8018070 con 0x7fd1080f5580 2021-11-20T05:38:58.867 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.864+0000 7fd0ef7fe700 1 -- 172.21.15.58:0/2905105300 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd0f8013410 con 0x7fd1080f5580 2021-11-20T05:38:58.867 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.864+0000 7fd0ef7fe700 1 -- 172.21.15.58:0/2905105300 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0f801c620 con 0x7fd1080f5580 2021-11-20T05:38:58.867 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.864+0000 7fd10d160700 1 -- 172.21.15.58:0/2905105300 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10810c540 con 0x7fd1080f5580 2021-11-20T05:38:58.867 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.864+0000 7fd10d160700 1 -- 172.21.15.58:0/2905105300 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd10810ca50 con 0x7fd1080f5580 2021-11-20T05:38:58.867 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.865+0000 7fd10d160700 1 -- 172.21.15.58:0/2905105300 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd108027a00 con 0x7fd1080f5580 2021-11-20T05:38:58.869 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.865+0000 7fd0ef7fe700 1 -- 172.21.15.58:0/2905105300 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd0f801d420 con 0x7fd1080f5580 2021-11-20T05:38:58.869 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.866+0000 7fd0ef7fe700 1 --2- 172.21.15.58:0/2905105300 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd0f005c2a0 0x7fd0f005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:38:58.869 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.866+0000 7fd0ef7fe700 1 -- 172.21.15.58:0/2905105300 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fd0f80782e0 con 0x7fd1080f5580 2021-11-20T05:38:58.869 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.866+0000 7fd10659c700 1 --2- 172.21.15.58:0/2905105300 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd0f005c2a0 0x7fd0f005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:38:58.870 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.866+0000 7fd10659c700 1 --2- 172.21.15.58:0/2905105300 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd0f005c2a0 0x7fd0f005e760 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fd0fc00b790 tx=0x7fd0fc008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:38:58.870 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:58.870+0000 7fd0ef7fe700 1 -- 172.21.15.58:0/2905105300 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd0f804aa40 con 0x7fd1080f5580 2021-11-20T05:38:59.061 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:59.066+0000 7fd10d160700 1 -- 172.21.15.58:0/2905105300 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1080fdd90 con 0x7fd1080f5580 2021-11-20T05:38:59.062 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:59.066+0000 7fd0ef7fe700 1 -- 172.21.15.58:0/2905105300 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2200 (secure 0 0 0) 0x7fd0f804e060 con 0x7fd1080f5580 2021-11-20T05:38:59.062 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:38:59.063 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:52.287814+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:38:59.064 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:59.068+0000 7fd10d160700 1 -- 172.21.15.58:0/2905105300 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd0f005c2a0 msgr2=0x7fd0f005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:59.064 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:59.068+0000 7fd10d160700 1 --2- 172.21.15.58:0/2905105300 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd0f005c2a0 0x7fd0f005e760 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fd0fc00b790 tx=0x7fd0fc008040).stop 2021-11-20T05:38:59.064 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:59.068+0000 7fd10d160700 1 -- 172.21.15.58:0/2905105300 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1080f5580 msgr2=0x7fd108060530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:38:59.065 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:59.068+0000 7fd10d160700 1 --2- 172.21.15.58:0/2905105300 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1080f5580 0x7fd108060530 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7fd0f8009f50 tx=0x7fd0f8009b30).stop 2021-11-20T05:38:59.065 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:59.068+0000 7fd10d160700 1 -- 172.21.15.58:0/2905105300 shutdown_connections 2021-11-20T05:38:59.065 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:59.068+0000 7fd10d160700 1 --2- 172.21.15.58:0/2905105300 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fd0f005c2a0 0x7fd0f005e760 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:59.065 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:59.068+0000 7fd10d160700 1 --2- 172.21.15.58:0/2905105300 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fd1080f5580 0x7fd108060530 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:59.066 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:59.068+0000 7fd10d160700 1 --2- 172.21.15.58:0/2905105300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1080f6820 0x7fd10810bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:38:59.066 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:59.068+0000 7fd10d160700 1 -- 172.21.15.58:0/2905105300 >> 172.21.15.58:0/2905105300 conn(0x7fd1080f0ce0 msgr2=0x7fd1080fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:38:59.066 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:59.068+0000 7fd10d160700 1 -- 172.21.15.58:0/2905105300 shutdown_connections 2021-11-20T05:38:59.066 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:38:59.068+0000 7fd10d160700 1 -- 172.21.15.58:0/2905105300 wait complete. 2021-11-20T05:38:59.067 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 100 2021-11-20T05:38:59.075 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:52.287814+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:38:59.076 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:rejoin 2021-11-20T05:38:59.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:59 smithi187 conmon[36358]: cluster 2021-11-20T05:38:58.308175+0000 mgr.smithi058.zagdlk (mgr.14164) 338 : cluster [DBG] pgmap v316: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:59.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:38:59 smithi187 conmon[36358]: audit 2021-11-20T05:38:59.067410+0000 mon.smithi058 (mon.0) 894 : audit [DBG] from='client.? 172.21.15.58:0/2905105300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:38:59.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:59 smithi058 conmon[33537]: cluster 2021-11-20T05:38:58.308175+0000 mgr.smithi058.zagdlk ( 2021-11-20T05:38:59.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:59 smithi058 conmon[33537]: mgr.14164) 338 : cluster [DBG] pgmap v316: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T05:38:59.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:59 smithi058 conmon[33537]: audit 2021-11-20T05:38:59.067410+0000 2021-11-20T05:38:59.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:38:59 smithi058 conmon[33537]: mon.smithi058 (mon.0) 894 : audit [DBG] from='client.? 172.21.15.58:0/2905105300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:00.077 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:39:00.212 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.214+0000 7f13210ee700 1 -- 172.21.15.58:0/199324671 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f131c0f6850 msgr2=0x7f131c0f6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:00.212 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.214+0000 7f13210ee700 1 --2- 172.21.15.58:0/199324671 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f131c0f6850 0x7f131c0f6cb0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f1304004660 tx=0x7f1304009b30).stop 2021-11-20T05:39:00.213 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.214+0000 7f13210ee700 1 -- 172.21.15.58:0/199324671 shutdown_connections 2021-11-20T05:39:00.213 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.214+0000 7f13210ee700 1 --2- 172.21.15.58:0/199324671 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f131c0f6850 0x7f131c0f6cb0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:00.213 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.214+0000 7f13210ee700 1 --2- 172.21.15.58:0/199324671 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f131c0f55b0 0x7f131c0f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:00.213 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.214+0000 7f13210ee700 1 -- 172.21.15.58:0/199324671 >> 172.21.15.58:0/199324671 conn(0x7f131c0f0d10 msgr2=0x7f131c0f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:00.213 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.214+0000 7f13210ee700 1 -- 172.21.15.58:0/199324671 shutdown_connections 2021-11-20T05:39:00.214 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.214+0000 7f13210ee700 1 -- 172.21.15.58:0/199324671 wait complete. 2021-11-20T05:39:00.214 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.215+0000 7f13210ee700 1 Processor -- start 2021-11-20T05:39:00.214 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.215+0000 7f13210ee700 1 -- start start 2021-11-20T05:39:00.214 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.215+0000 7f13210ee700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f131c0f55b0 0x7f131c0604b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:00.215 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.215+0000 7f13210ee700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f131c0f6850 0x7f131c10bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:00.215 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.215+0000 7f13210ee700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c10c170 con 0x7f131c0f6850 2021-11-20T05:39:00.215 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.215+0000 7f13210ee700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c10c2b0 con 0x7f131c0f55b0 2021-11-20T05:39:00.215 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.215+0000 7f131a59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f131c0f6850 0x7f131c10bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:00.216 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.215+0000 7f131a59c700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f131c0f6850 0x7f131c10bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42476/0 (socket says 172.21.15.58:42476) 2021-11-20T05:39:00.216 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.215+0000 7f131a59c700 1 -- 172.21.15.58:0/922354552 learned_addr learned my addr 172.21.15.58:0/922354552 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:39:00.216 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.215+0000 7f131ad9d700 1 --2- 172.21.15.58:0/922354552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f131c0f55b0 0x7f131c0604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:00.216 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.215+0000 7f131a59c700 1 -- 172.21.15.58:0/922354552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f131c0f55b0 msgr2=0x7f131c0604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:00.216 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.215+0000 7f131a59c700 1 --2- 172.21.15.58:0/922354552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f131c0f55b0 0x7f131c0604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:00.217 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.215+0000 7f131a59c700 1 -- 172.21.15.58:0/922354552 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1304005040 con 0x7f131c0f6850 2021-11-20T05:39:00.217 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.215+0000 7f131a59c700 1 --2- 172.21.15.58:0/922354552 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f131c0f6850 0x7f131c10bc80 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f1304009830 tx=0x7f1304009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:00.217 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.216+0000 7f13137fe700 1 -- 172.21.15.58:0/922354552 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1304018070 con 0x7f131c0f6850 2021-11-20T05:39:00.217 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.216+0000 7f13137fe700 1 -- 172.21.15.58:0/922354552 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1304013410 con 0x7f131c0f6850 2021-11-20T05:39:00.218 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.216+0000 7f13210ee700 1 -- 172.21.15.58:0/922354552 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131c10c4b0 con 0x7f131c0f6850 2021-11-20T05:39:00.218 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.216+0000 7f13137fe700 1 -- 172.21.15.58:0/922354552 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f130401c620 con 0x7f131c0f6850 2021-11-20T05:39:00.218 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.216+0000 7f13210ee700 1 -- 172.21.15.58:0/922354552 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f131c10c9e0 con 0x7f131c0f6850 2021-11-20T05:39:00.219 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.217+0000 7f13210ee700 1 -- 172.21.15.58:0/922354552 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f131c046330 con 0x7f131c0f6850 2021-11-20T05:39:00.220 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.217+0000 7f13137fe700 1 -- 172.21.15.58:0/922354552 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f130401c8e0 con 0x7f131c0f6850 2021-11-20T05:39:00.220 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.217+0000 7f13137fe700 1 --2- 172.21.15.58:0/922354552 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f130805c2a0 0x7f130805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:00.220 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.217+0000 7f13137fe700 1 -- 172.21.15.58:0/922354552 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f1304078320 con 0x7f131c0f6850 2021-11-20T05:39:00.221 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.218+0000 7f131ad9d700 1 --2- 172.21.15.58:0/922354552 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f130805c2a0 0x7f130805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:00.221 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.218+0000 7f131ad9d700 1 --2- 172.21.15.58:0/922354552 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f130805c2a0 0x7f130805e760 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f131c0f66b0 tx=0x7f130c009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:00.221 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.221+0000 7f13137fe700 1 -- 172.21.15.58:0/922354552 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f130404aa20 con 0x7f131c0f6850 2021-11-20T05:39:00.414 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.418+0000 7f13210ee700 1 -- 172.21.15.58:0/922354552 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f131c027a00 con 0x7f131c0f6850 2021-11-20T05:39:00.414 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.418+0000 7f13137fe700 1 -- 172.21.15.58:0/922354552 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2200 (secure 0 0 0) 0x7f130404e040 con 0x7f131c0f6850 2021-11-20T05:39:00.415 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:39:00.415 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:52.287814+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:39:00.416 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.420+0000 7f13210ee700 1 -- 172.21.15.58:0/922354552 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f130805c2a0 msgr2=0x7f130805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:00.417 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.420+0000 7f13210ee700 1 --2- 172.21.15.58:0/922354552 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f130805c2a0 0x7f130805e760 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f131c0f66b0 tx=0x7f130c009040).stop 2021-11-20T05:39:00.417 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.420+0000 7f13210ee700 1 -- 172.21.15.58:0/922354552 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f131c0f6850 msgr2=0x7f131c10bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:00.417 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.420+0000 7f13210ee700 1 --2- 172.21.15.58:0/922354552 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f131c0f6850 0x7f131c10bc80 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f1304009830 tx=0x7f1304009b30).stop 2021-11-20T05:39:00.417 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.420+0000 7f13210ee700 1 -- 172.21.15.58:0/922354552 shutdown_connections 2021-11-20T05:39:00.417 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.420+0000 7f13210ee700 1 --2- 172.21.15.58:0/922354552 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f130805c2a0 0x7f130805e760 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:00.418 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.420+0000 7f13210ee700 1 --2- 172.21.15.58:0/922354552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f131c0f55b0 0x7f131c0604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:00.418 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.420+0000 7f13210ee700 1 --2- 172.21.15.58:0/922354552 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f131c0f6850 0x7f131c10bc80 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:00.418 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.420+0000 7f13210ee700 1 -- 172.21.15.58:0/922354552 >> 172.21.15.58:0/922354552 conn(0x7f131c0f0d10 msgr2=0x7f131c0fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:00.418 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.420+0000 7f13210ee700 1 -- 172.21.15.58:0/922354552 shutdown_connections 2021-11-20T05:39:00.419 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:00.420+0000 7f13210ee700 1 -- 172.21.15.58:0/922354552 wait complete. 2021-11-20T05:39:00.419 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 100 2021-11-20T05:39:00.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:52.287814+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:39:00.428 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:rejoin 2021-11-20T05:39:00.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:00 smithi058 conmon[33537]: audit 2021-11-20T05:39:00.394863+0000 mon.smithi058 ( 2021-11-20T05:39:00.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:00 smithi058 conmon[33537]: mon.0) 895 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:39:00.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:00 smithi058 conmon[33537]: audit 2021-11-20T05:39:00.398454+0000 mon.smithi058 (mon.0) 896 : audit 2021-11-20T05:39:00.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:00 smithi058 conmon[33537]: [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:39:00.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:00 smithi058 conmon[33537]: audit 2021-11-20T05:39:00 2021-11-20T05:39:00.716 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:00 smithi058 conmon[33537]: .420078+0000 mon.smithi058 (mon.0) 897 : audit [DBG] from='client.? 172.21.15.58:0/922354552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:00.943 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:00 smithi187 conmon[36358]: audit 2021-11-20T05:39:00.394863+0000 mon.smithi058 (mon.0) 895 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/trash_purge_schedule"}]: dispatch 2021-11-20T05:39:00.943 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:00 smithi187 conmon[36358]: audit 2021-11-20T05:39:00 2021-11-20T05:39:00.943 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:00 smithi187 conmon[36358]: .398454+0000 mon.smithi058 (mon.0) 896 : audit [INF] from='mgr.14164 172.21.15.58:0/3326958300' entity='mgr.smithi058.zagdlk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi058.zagdlk/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:39:00.943 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:00 smithi187 conmon[36358]: audit 2021-11-20T05:39:00.420078+0000 mon.smithi058 (mon.0) 897 : audit [DBG] from='client.? 172.21.15.58:0/922354552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:01.430 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:39:01.567 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.567+0000 7f10efc32700 1 -- 172.21.15.58:0/3210233030 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10e80f55a0 msgr2=0x7f10e80f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:01.567 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.567+0000 7f10efc32700 1 --2- 172.21.15.58:0/3210233030 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10e80f55a0 0x7f10e80f59c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f10e4004660 tx=0x7f10e4009b30).stop 2021-11-20T05:39:01.567 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.567+0000 7f10efc32700 1 -- 172.21.15.58:0/3210233030 shutdown_connections 2021-11-20T05:39:01.567 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.567+0000 7f10efc32700 1 --2- 172.21.15.58:0/3210233030 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10e80f6840 0x7f10e80f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:01.568 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.567+0000 7f10efc32700 1 --2- 172.21.15.58:0/3210233030 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10e80f55a0 0x7f10e80f59c0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:01.568 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.567+0000 7f10efc32700 1 -- 172.21.15.58:0/3210233030 >> 172.21.15.58:0/3210233030 conn(0x7f10e80f0ce0 msgr2=0x7f10e80f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:01.568 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.567+0000 7f10efc32700 1 -- 172.21.15.58:0/3210233030 shutdown_connections 2021-11-20T05:39:01.568 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.567+0000 7f10efc32700 1 -- 172.21.15.58:0/3210233030 wait complete. 2021-11-20T05:39:01.569 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.568+0000 7f10efc32700 1 Processor -- start 2021-11-20T05:39:01.569 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.568+0000 7f10efc32700 1 -- start start 2021-11-20T05:39:01.570 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.568+0000 7f10efc32700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10e80f55a0 0x7f10e805ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:01.570 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.568+0000 7f10efc32700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10e80f6840 0x7f10e810bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:01.570 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.568+0000 7f10efc32700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e810c1d0 con 0x7f10e80f55a0 2021-11-20T05:39:01.570 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.568+0000 7f10efc32700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e810c310 con 0x7f10e80f6840 2021-11-20T05:39:01.570 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.568+0000 7f10ed9ce700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10e80f55a0 0x7f10e805ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:01.571 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.568+0000 7f10ed9ce700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10e80f55a0 0x7f10e805ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42484/0 (socket says 172.21.15.58:42484) 2021-11-20T05:39:01.571 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.568+0000 7f10ed9ce700 1 -- 172.21.15.58:0/4052352974 learned_addr learned my addr 172.21.15.58:0/4052352974 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:39:01.571 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.568+0000 7f10ed1cd700 1 --2- 172.21.15.58:0/4052352974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10e80f6840 0x7f10e810bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:01.571 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.568+0000 7f10ed9ce700 1 -- 172.21.15.58:0/4052352974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10e80f6840 msgr2=0x7f10e810bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:01.572 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.568+0000 7f10ed9ce700 1 --2- 172.21.15.58:0/4052352974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10e80f6840 0x7f10e810bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:01.572 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.569+0000 7f10ed9ce700 1 -- 172.21.15.58:0/4052352974 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e4005040 con 0x7f10e80f55a0 2021-11-20T05:39:01.572 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.569+0000 7f10ed9ce700 1 --2- 172.21.15.58:0/4052352974 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10e80f55a0 0x7f10e805ff00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f10e4009f50 tx=0x7f10e4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:01.572 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.569+0000 7f10da7fc700 1 -- 172.21.15.58:0/4052352974 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10e4018070 con 0x7f10e80f55a0 2021-11-20T05:39:01.572 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.569+0000 7f10da7fc700 1 -- 172.21.15.58:0/4052352974 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f10e4013410 con 0x7f10e80f55a0 2021-11-20T05:39:01.573 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.569+0000 7f10efc32700 1 -- 172.21.15.58:0/4052352974 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e810c510 con 0x7f10e80f55a0 2021-11-20T05:39:01.573 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.569+0000 7f10da7fc700 1 -- 172.21.15.58:0/4052352974 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10e401c620 con 0x7f10e80f55a0 2021-11-20T05:39:01.573 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.569+0000 7f10efc32700 1 -- 172.21.15.58:0/4052352974 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e810ca40 con 0x7f10e80f55a0 2021-11-20T05:39:01.573 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.570+0000 7f10efc32700 1 -- 172.21.15.58:0/4052352974 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e8046330 con 0x7f10e80f55a0 2021-11-20T05:39:01.573 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.570+0000 7f10da7fc700 1 -- 172.21.15.58:0/4052352974 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f10e401d420 con 0x7f10e80f55a0 2021-11-20T05:39:01.574 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.570+0000 7f10da7fc700 1 --2- 172.21.15.58:0/4052352974 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f10d405c2a0 0x7f10d405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:01.574 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.571+0000 7f10da7fc700 1 -- 172.21.15.58:0/4052352974 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f10e4078290 con 0x7f10e80f55a0 2021-11-20T05:39:01.574 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.571+0000 7f10ed1cd700 1 --2- 172.21.15.58:0/4052352974 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f10d405c2a0 0x7f10d405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:01.574 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.571+0000 7f10ed1cd700 1 --2- 172.21.15.58:0/4052352974 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f10d405c2a0 0x7f10d405e760 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f10dc007fd0 tx=0x7f10dc009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:01.575 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.574+0000 7f10da7fc700 1 -- 172.21.15.58:0/4052352974 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f10e404a9f0 con 0x7f10e80f55a0 2021-11-20T05:39:01.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:01 smithi058 conmon[33537]: cluster 2021-11-20T05:39:00.308607+0000 mgr.smithi058.zagdlk (mgr.14164) 339 : cluster [DBG] pgmap v317: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T05:39:01.769 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.773+0000 7f10efc32700 1 -- 172.21.15.58:0/4052352974 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10e8027a00 con 0x7f10e80f55a0 2021-11-20T05:39:01.770 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.773+0000 7f10da7fc700 1 -- 172.21.15.58:0/4052352974 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2200 (secure 0 0 0) 0x7f10e404e010 con 0x7f10e80f55a0 2021-11-20T05:39:01.771 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:39:01.771 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:52.287814+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:39:01.772 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.775+0000 7f10efc32700 1 -- 172.21.15.58:0/4052352974 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f10d405c2a0 msgr2=0x7f10d405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:01.772 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.775+0000 7f10efc32700 1 --2- 172.21.15.58:0/4052352974 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f10d405c2a0 0x7f10d405e760 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f10dc007fd0 tx=0x7f10dc009040).stop 2021-11-20T05:39:01.772 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.775+0000 7f10efc32700 1 -- 172.21.15.58:0/4052352974 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10e80f55a0 msgr2=0x7f10e805ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:01.773 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.775+0000 7f10efc32700 1 --2- 172.21.15.58:0/4052352974 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10e80f55a0 0x7f10e805ff00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f10e4009f50 tx=0x7f10e4009b30).stop 2021-11-20T05:39:01.773 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.775+0000 7f10efc32700 1 -- 172.21.15.58:0/4052352974 shutdown_connections 2021-11-20T05:39:01.773 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.775+0000 7f10efc32700 1 --2- 172.21.15.58:0/4052352974 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f10d405c2a0 0x7f10d405e760 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:01.773 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.775+0000 7f10efc32700 1 --2- 172.21.15.58:0/4052352974 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f10e80f55a0 0x7f10e805ff00 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:01.774 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.775+0000 7f10efc32700 1 --2- 172.21.15.58:0/4052352974 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10e80f6840 0x7f10e810bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:01.774 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.775+0000 7f10efc32700 1 -- 172.21.15.58:0/4052352974 >> 172.21.15.58:0/4052352974 conn(0x7f10e80f0ce0 msgr2=0x7f10e80fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:01.774 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.775+0000 7f10efc32700 1 -- 172.21.15.58:0/4052352974 shutdown_connections 2021-11-20T05:39:01.774 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:01.775+0000 7f10efc32700 1 -- 172.21.15.58:0/4052352974 wait complete. 2021-11-20T05:39:01.774 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 100 2021-11-20T05:39:01.785 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:52.287814+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:39:01.786 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:rejoin 2021-11-20T05:39:01.942 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:01 smithi187 conmon[36358]: cluster 2021-11-20T05:39:00.308607+0000 mgr.smithi058.zagdlk (mgr.14164) 339 : cluster [DBG] pgmap v317: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T05:39:02.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:02 smithi058 conmon[33537]: audit 2021-11-20T05:39:01.775506+0000 mon.smithi058 (mon.0) 898 2021-11-20T05:39:02.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:02 smithi058 conmon[33537]: : audit [DBG] from='client.? 172.21.15.58:0/4052352974' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:02.787 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:39:02.921 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.923+0000 7f44ed99b700 1 -- 172.21.15.58:0/2220941016 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44e80f97f0 msgr2=0x7f44e80fbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:02.922 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.923+0000 7f44ed99b700 1 --2- 172.21.15.58:0/2220941016 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44e80f97f0 0x7f44e80fbc40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f44d0004660 tx=0x7f44d0009b30).stop 2021-11-20T05:39:02.922 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.923+0000 7f44ed99b700 1 -- 172.21.15.58:0/2220941016 shutdown_connections 2021-11-20T05:39:02.922 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.923+0000 7f44ed99b700 1 --2- 172.21.15.58:0/2220941016 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44e80f97f0 0x7f44e80fbc40 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:02.922 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.923+0000 7f44ed99b700 1 --2- 172.21.15.58:0/2220941016 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44e8060690 0x7f44e80f9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:02.923 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.923+0000 7f44ed99b700 1 -- 172.21.15.58:0/2220941016 >> 172.21.15.58:0/2220941016 conn(0x7f44e80f0ce0 msgr2=0x7f44e80f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:02.923 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.923+0000 7f44ed99b700 1 -- 172.21.15.58:0/2220941016 shutdown_connections 2021-11-20T05:39:02.923 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.923+0000 7f44ed99b700 1 -- 172.21.15.58:0/2220941016 wait complete. 2021-11-20T05:39:02.923 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.924+0000 7f44ed99b700 1 Processor -- start 2021-11-20T05:39:02.924 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.924+0000 7f44ed99b700 1 -- start start 2021-11-20T05:39:02.924 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.924+0000 7f44ed99b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44e8060690 0x7f44e80f6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:02.924 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.924+0000 7f44ed99b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44e80f97f0 0x7f44e80f5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:02.924 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.924+0000 7f44ed99b700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80f7150 con 0x7f44e8060690 2021-11-20T05:39:02.924 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.924+0000 7f44ed99b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80f56f0 con 0x7f44e80f97f0 2021-11-20T05:39:02.925 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.924+0000 7f44e6ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44e8060690 0x7f44e80f6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:02.925 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.925+0000 7f44e6ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44e8060690 0x7f44e80f6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42494/0 (socket says 172.21.15.58:42494) 2021-11-20T05:39:02.925 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.925+0000 7f44e6ffd700 1 -- 172.21.15.58:0/3659373296 learned_addr learned my addr 172.21.15.58:0/3659373296 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:39:02.925 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.925+0000 7f44e67fc700 1 --2- 172.21.15.58:0/3659373296 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44e80f97f0 0x7f44e80f5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:02.926 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.925+0000 7f44e6ffd700 1 -- 172.21.15.58:0/3659373296 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44e80f97f0 msgr2=0x7f44e80f5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:02.927 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.925+0000 7f44e6ffd700 1 --2- 172.21.15.58:0/3659373296 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44e80f97f0 0x7f44e80f5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:02.927 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.925+0000 7f44e6ffd700 1 -- 172.21.15.58:0/3659373296 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d0005040 con 0x7f44e8060690 2021-11-20T05:39:02.927 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.925+0000 7f44e6ffd700 1 --2- 172.21.15.58:0/3659373296 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44e8060690 0x7f44e80f6ba0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f44e0007fd0 tx=0x7f44e0004ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:02.927 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.925+0000 7f44df7fe700 1 -- 172.21.15.58:0/3659373296 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44e000b5a0 con 0x7f44e8060690 2021-11-20T05:39:02.928 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.925+0000 7f44ed99b700 1 -- 172.21.15.58:0/3659373296 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e80f5970 con 0x7f44e8060690 2021-11-20T05:39:02.928 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.925+0000 7f44df7fe700 1 -- 172.21.15.58:0/3659373296 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44e000f040 con 0x7f44e8060690 2021-11-20T05:39:02.928 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.925+0000 7f44df7fe700 1 -- 172.21.15.58:0/3659373296 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44e0013b00 con 0x7f44e8060690 2021-11-20T05:39:02.928 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.925+0000 7f44ed99b700 1 -- 172.21.15.58:0/3659373296 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e80f5f30 con 0x7f44e8060690 2021-11-20T05:39:02.929 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.926+0000 7f44ed99b700 1 -- 172.21.15.58:0/3659373296 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e8027a00 con 0x7f44e8060690 2021-11-20T05:39:02.930 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.926+0000 7f44df7fe700 1 -- 172.21.15.58:0/3659373296 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f44e001a030 con 0x7f44e8060690 2021-11-20T05:39:02.930 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.927+0000 7f44df7fe700 1 --2- 172.21.15.58:0/3659373296 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44d405c2a0 0x7f44d405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:02.930 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.927+0000 7f44df7fe700 1 -- 172.21.15.58:0/3659373296 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f44e0076190 con 0x7f44e8060690 2021-11-20T05:39:02.930 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.927+0000 7f44e67fc700 1 --2- 172.21.15.58:0/3659373296 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44d405c2a0 0x7f44d405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:02.931 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.927+0000 7f44e67fc700 1 --2- 172.21.15.58:0/3659373296 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44d405c2a0 0x7f44d405e760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f44d0004e30 tx=0x7f44d0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:02.931 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:02.931+0000 7f44df7fe700 1 -- 172.21.15.58:0/3659373296 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44e00489a0 con 0x7f44e8060690 2021-11-20T05:39:02.942 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:02 smithi187 conmon[36358]: audit 2021-11-20T05:39:01.775506+0000 mon.smithi058 (mon.0) 898 : audit [DBG] from='client.? 172.21.15.58:0/4052352974' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:03.124 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:03.127+0000 7f44ed99b700 1 -- 172.21.15.58:0/3659373296 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44e80fee20 con 0x7f44e8060690 2021-11-20T05:39:03.125 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:03.128+0000 7f44df7fe700 1 -- 172.21.15.58:0/3659373296 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2200 (secure 0 0 0) 0x7f44e0018070 con 0x7f44e8060690 2021-11-20T05:39:03.126 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:39:03.127 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:52.287814+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:39:03.129 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:03.129+0000 7f44ed99b700 1 -- 172.21.15.58:0/3659373296 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44d405c2a0 msgr2=0x7f44d405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:03.129 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:03.129+0000 7f44ed99b700 1 --2- 172.21.15.58:0/3659373296 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44d405c2a0 0x7f44d405e760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f44d0004e30 tx=0x7f44d0009b30).stop 2021-11-20T05:39:03.130 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:03.129+0000 7f44ed99b700 1 -- 172.21.15.58:0/3659373296 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44e8060690 msgr2=0x7f44e80f6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:03.130 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:03.129+0000 7f44ed99b700 1 --2- 172.21.15.58:0/3659373296 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44e8060690 0x7f44e80f6ba0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f44e0007fd0 tx=0x7f44e0004ee0).stop 2021-11-20T05:39:03.131 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:03.130+0000 7f44ed99b700 1 -- 172.21.15.58:0/3659373296 shutdown_connections 2021-11-20T05:39:03.131 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:03.130+0000 7f44ed99b700 1 --2- 172.21.15.58:0/3659373296 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f44d405c2a0 0x7f44d405e760 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:03.132 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:03.130+0000 7f44ed99b700 1 --2- 172.21.15.58:0/3659373296 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f44e8060690 0x7f44e80f6ba0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:03.132 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:03.130+0000 7f44ed99b700 1 --2- 172.21.15.58:0/3659373296 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44e80f97f0 0x7f44e80f5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:03.133 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:03.130+0000 7f44ed99b700 1 -- 172.21.15.58:0/3659373296 >> 172.21.15.58:0/3659373296 conn(0x7f44e80f0ce0 msgr2=0x7f44e80f30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:03.133 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:03.130+0000 7f44ed99b700 1 -- 172.21.15.58:0/3659373296 shutdown_connections 2021-11-20T05:39:03.134 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:03.130+0000 7f44ed99b700 1 -- 172.21.15.58:0/3659373296 wait complete. 2021-11-20T05:39:03.134 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 100 2021-11-20T05:39:03.138 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:52.287814+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:39:03.138 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:rejoin 2021-11-20T05:39:03.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:03 smithi058 conmon[33537]: cluster 2021-11-20T05:39:02.309117+0000 mgr.smithi058.zagdlk (mgr.14164) 340 : cluster [DBG] pgmap v318: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:39:03.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:03 smithi058 conmon[33537]: audit 2021-11-20T05:39:03.130013+0000 mon.smithi058 (mon.0) 899 : audit [DBG] from='client.? 172.21.15.58:0/3659373296' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:03.942 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:03 smithi187 conmon[36358]: cluster 2021-11-20T05:39:02.309117+0000 mgr.smithi058.zagdlk (mgr.14164) 340 : cluster [DBG] pgmap v318: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T05:39:03.943 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:03 smithi187 conmon[36358]: audit 2021-11-20T05:39:03.130013+0000 mon.smithi058 (mon.0) 899 : audit [DBG] from='client.? 172.21.15.58:0/3659373296' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:04.139 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:39:04.274 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.276+0000 7f5c41ffa700 1 -- 172.21.15.58:0/4234458636 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5c3c0f65d0 msgr2=0x7f5c3c0f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:04.275 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.276+0000 7f5c41ffa700 1 --2- 172.21.15.58:0/4234458636 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5c3c0f65d0 0x7f5c3c0f69f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f5c30004660 tx=0x7f5c30009b30).stop 2021-11-20T05:39:04.275 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.276+0000 7f5c41ffa700 1 -- 172.21.15.58:0/4234458636 shutdown_connections 2021-11-20T05:39:04.275 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.276+0000 7f5c41ffa700 1 --2- 172.21.15.58:0/4234458636 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c3c0f7870 0x7f5c3c0f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:04.275 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.276+0000 7f5c41ffa700 1 --2- 172.21.15.58:0/4234458636 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5c3c0f65d0 0x7f5c3c0f69f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:04.275 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.276+0000 7f5c41ffa700 1 -- 172.21.15.58:0/4234458636 >> 172.21.15.58:0/4234458636 conn(0x7f5c3c0f1d10 msgr2=0x7f5c3c0f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:04.276 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.276+0000 7f5c41ffa700 1 -- 172.21.15.58:0/4234458636 shutdown_connections 2021-11-20T05:39:04.276 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.276+0000 7f5c41ffa700 1 -- 172.21.15.58:0/4234458636 wait complete. 2021-11-20T05:39:04.276 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.277+0000 7f5c41ffa700 1 Processor -- start 2021-11-20T05:39:04.276 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.277+0000 7f5c41ffa700 1 -- start start 2021-11-20T05:39:04.276 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.277+0000 7f5c41ffa700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5c3c0f65d0 0x7f5c3c106da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:04.277 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.277+0000 7f5c41ffa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c3c0f7870 0x7f5c3c107290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:04.277 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.277+0000 7f5c41ffa700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c107890 con 0x7f5c3c0f65d0 2021-11-20T05:39:04.277 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.277+0000 7f5c41ffa700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c1080d0 con 0x7f5c3c0f7870 2021-11-20T05:39:04.277 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.277+0000 7f5c40ff8700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5c3c0f65d0 0x7f5c3c106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:04.278 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.277+0000 7f5c40ff8700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5c3c0f65d0 0x7f5c3c106da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42504/0 (socket says 172.21.15.58:42504) 2021-11-20T05:39:04.278 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.277+0000 7f5c40ff8700 1 -- 172.21.15.58:0/1952185178 learned_addr learned my addr 172.21.15.58:0/1952185178 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:39:04.278 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.278+0000 7f5c40ff8700 1 -- 172.21.15.58:0/1952185178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c3c0f7870 msgr2=0x7f5c3c107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:04.278 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.278+0000 7f5c40ff8700 1 --2- 172.21.15.58:0/1952185178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c3c0f7870 0x7f5c3c107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:39:04.279 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.278+0000 7f5c40ff8700 1 -- 172.21.15.58:0/1952185178 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c30005040 con 0x7f5c3c0f65d0 2021-11-20T05:39:04.279 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.278+0000 7f5c40ff8700 1 --2- 172.21.15.58:0/1952185178 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5c3c0f65d0 0x7f5c3c106da0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f5c30004e30 tx=0x7f5c30009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:04.279 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.278+0000 7f5c397fa700 1 -- 172.21.15.58:0/1952185178 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c30018070 con 0x7f5c3c0f65d0 2021-11-20T05:39:04.279 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.278+0000 7f5c397fa700 1 -- 172.21.15.58:0/1952185178 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c30013410 con 0x7f5c3c0f65d0 2021-11-20T05:39:04.279 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.278+0000 7f5c41ffa700 1 -- 172.21.15.58:0/1952185178 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3c108350 con 0x7f5c3c0f65d0 2021-11-20T05:39:04.280 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.278+0000 7f5c397fa700 1 -- 172.21.15.58:0/1952185178 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c3001c3c0 con 0x7f5c3c0f65d0 2021-11-20T05:39:04.280 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.278+0000 7f5c41ffa700 1 -- 172.21.15.58:0/1952185178 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3c108830 con 0x7f5c3c0f65d0 2021-11-20T05:39:04.281 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.279+0000 7f5c41ffa700 1 -- 172.21.15.58:0/1952185178 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3c0f59c0 con 0x7f5c3c0f65d0 2021-11-20T05:39:04.281 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.280+0000 7f5c397fa700 1 -- 172.21.15.58:0/1952185178 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5c3001d420 con 0x7f5c3c0f65d0 2021-11-20T05:39:04.281 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.280+0000 7f5c397fa700 1 --2- 172.21.15.58:0/1952185178 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5c2c05c300 0x7f5c2c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:04.282 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.280+0000 7f5c397fa700 1 -- 172.21.15.58:0/1952185178 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f5c30078510 con 0x7f5c3c0f65d0 2021-11-20T05:39:04.282 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.280+0000 7f5c3bfff700 1 --2- 172.21.15.58:0/1952185178 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5c2c05c300 0x7f5c2c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:04.282 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.281+0000 7f5c3bfff700 1 --2- 172.21.15.58:0/1952185178 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5c2c05c300 0x7f5c2c05e7c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f5c3c106b30 tx=0x7f5c280099e0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:04.282 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.284+0000 7f5c397fa700 1 -- 172.21.15.58:0/1952185178 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c3004aca0 con 0x7f5c3c0f65d0 2021-11-20T05:39:04.479 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.482+0000 7f5c41ffa700 1 -- 172.21.15.58:0/1952185178 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c3c045ad0 con 0x7f5c3c0f65d0 2021-11-20T05:39:04.479 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.483+0000 7f5c397fa700 1 -- 172.21.15.58:0/1952185178 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2200 (secure 0 0 0) 0x7f5c3004e2c0 con 0x7f5c3c0f65d0 2021-11-20T05:39:04.479 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:39:04.480 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:52.287814+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:39:04.481 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.485+0000 7f5c41ffa700 1 -- 172.21.15.58:0/1952185178 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5c2c05c300 msgr2=0x7f5c2c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:04.482 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.485+0000 7f5c41ffa700 1 --2- 172.21.15.58:0/1952185178 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5c2c05c300 0x7f5c2c05e7c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f5c3c106b30 tx=0x7f5c280099e0).stop 2021-11-20T05:39:04.482 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.485+0000 7f5c41ffa700 1 -- 172.21.15.58:0/1952185178 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5c3c0f65d0 msgr2=0x7f5c3c106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:04.482 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.485+0000 7f5c41ffa700 1 --2- 172.21.15.58:0/1952185178 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5c3c0f65d0 0x7f5c3c106da0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f5c30004e30 tx=0x7f5c30009b30).stop 2021-11-20T05:39:04.482 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.485+0000 7f5c41ffa700 1 -- 172.21.15.58:0/1952185178 shutdown_connections 2021-11-20T05:39:04.482 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.485+0000 7f5c41ffa700 1 --2- 172.21.15.58:0/1952185178 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f5c2c05c300 0x7f5c2c05e7c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:04.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.485+0000 7f5c41ffa700 1 --2- 172.21.15.58:0/1952185178 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f5c3c0f65d0 0x7f5c3c106da0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:04.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.485+0000 7f5c41ffa700 1 --2- 172.21.15.58:0/1952185178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c3c0f7870 0x7f5c3c107290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:39:04.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.485+0000 7f5c41ffa700 1 -- 172.21.15.58:0/1952185178 >> 172.21.15.58:0/1952185178 conn(0x7f5c3c0f1d10 msgr2=0x7f5c3c0fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:04.483 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.485+0000 7f5c41ffa700 1 -- 172.21.15.58:0/1952185178 shutdown_connections 2021-11-20T05:39:04.484 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:04.485+0000 7f5c41ffa700 1 -- 172.21.15.58:0/1952185178 wait complete. 2021-11-20T05:39:04.484 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 100 2021-11-20T05:39:04.493 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:52.287814+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:39:04.493 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:rejoin 2021-11-20T05:39:05.494 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:39:05.629 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.631+0000 7fce7b59e700 1 -- 172.21.15.58:0/334513760 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce7c0f6a60 msgr2=0x7fce7c0f6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:05.630 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.631+0000 7fce7b59e700 1 --2- 172.21.15.58:0/334513760 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce7c0f6a60 0x7fce7c0f6e80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7fce6c004660 tx=0x7fce6c009b30).stop 2021-11-20T05:39:05.630 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.632+0000 7fce7b59e700 1 -- 172.21.15.58:0/334513760 shutdown_connections 2021-11-20T05:39:05.630 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.632+0000 7fce7b59e700 1 --2- 172.21.15.58:0/334513760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce7c0f7400 0x7fce7c0f5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:05.631 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.632+0000 7fce7b59e700 1 --2- 172.21.15.58:0/334513760 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce7c0f6a60 0x7fce7c0f6e80 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:05.631 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.632+0000 7fce7b59e700 1 -- 172.21.15.58:0/334513760 >> 172.21.15.58:0/334513760 conn(0x7fce7c0f0360 msgr2=0x7fce7c0f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:05.631 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.632+0000 7fce7b59e700 1 -- 172.21.15.58:0/334513760 shutdown_connections 2021-11-20T05:39:05.631 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.632+0000 7fce7b59e700 1 -- 172.21.15.58:0/334513760 wait complete. 2021-11-20T05:39:05.632 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.632+0000 7fce7b59e700 1 Processor -- start 2021-11-20T05:39:05.632 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.632+0000 7fce7b59e700 1 -- start start 2021-11-20T05:39:05.632 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.632+0000 7fce7b59e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce7c0f6a60 0x7fce7c0fd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:05.632 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.632+0000 7fce7b59e700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce7c0f7400 0x7fce7c0fbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:05.633 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.632+0000 7fce7b59e700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7c0fc120 con 0x7fce7c0f7400 2021-11-20T05:39:05.633 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.632+0000 7fce7b59e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7c0fc260 con 0x7fce7c0f6a60 2021-11-20T05:39:05.633 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.632+0000 7fce79d9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce7c0f7400 0x7fce7c0fbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:05.633 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.632+0000 7fce79d9b700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce7c0f7400 0x7fce7c0fbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42518/0 (socket says 172.21.15.58:42518) 2021-11-20T05:39:05.633 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.632+0000 7fce79d9b700 1 -- 172.21.15.58:0/1819818021 learned_addr learned my addr 172.21.15.58:0/1819818021 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:39:05.634 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.633+0000 7fce7a59c700 1 --2- 172.21.15.58:0/1819818021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce7c0f6a60 0x7fce7c0fd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:05.634 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.633+0000 7fce79d9b700 1 -- 172.21.15.58:0/1819818021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce7c0f6a60 msgr2=0x7fce7c0fd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:05.634 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.633+0000 7fce79d9b700 1 --2- 172.21.15.58:0/1819818021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce7c0f6a60 0x7fce7c0fd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:05.634 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.633+0000 7fce79d9b700 1 -- 172.21.15.58:0/1819818021 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c005040 con 0x7fce7c0f7400 2021-11-20T05:39:05.635 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.633+0000 7fce79d9b700 1 --2- 172.21.15.58:0/1819818021 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce7c0f7400 0x7fce7c0fbbc0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7fce70002700 tx=0x7fce70009d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:05.635 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.633+0000 7fce6affd700 1 -- 172.21.15.58:0/1819818021 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce700079c0 con 0x7fce7c0f7400 2021-11-20T05:39:05.635 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.633+0000 7fce6affd700 1 -- 172.21.15.58:0/1819818021 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce7000f040 con 0x7fce7c0f7400 2021-11-20T05:39:05.635 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.633+0000 7fce7b59e700 1 -- 172.21.15.58:0/1819818021 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce7c0fc540 con 0x7fce7c0f7400 2021-11-20T05:39:05.636 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.633+0000 7fce6affd700 1 -- 172.21.15.58:0/1819818021 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce70014740 con 0x7fce7c0f7400 2021-11-20T05:39:05.636 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.633+0000 7fce7b59e700 1 -- 172.21.15.58:0/1819818021 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce7c10edc0 con 0x7fce7c0f7400 2021-11-20T05:39:05.637 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.634+0000 7fce7b59e700 1 -- 172.21.15.58:0/1819818021 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce7c045ad0 con 0x7fce7c0f7400 2021-11-20T05:39:05.637 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.634+0000 7fce6affd700 1 -- 172.21.15.58:0/1819818021 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fce7001b060 con 0x7fce7c0f7400 2021-11-20T05:39:05.638 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.635+0000 7fce6affd700 1 --2- 172.21.15.58:0/1819818021 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fce6405c300 0x7fce6405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:05.638 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.635+0000 7fce6affd700 1 -- 172.21.15.58:0/1819818021 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fce700762e0 con 0x7fce7c0f7400 2021-11-20T05:39:05.638 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.635+0000 7fce7a59c700 1 --2- 172.21.15.58:0/1819818021 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fce6405c300 0x7fce6405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:05.639 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.635+0000 7fce7a59c700 1 --2- 172.21.15.58:0/1819818021 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fce6405c300 0x7fce6405e7c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fce7c0fd4d0 tx=0x7fce6c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:05.639 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.639+0000 7fce6affd700 1 -- 172.21.15.58:0/1819818021 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce700489e0 con 0x7fce7c0f7400 2021-11-20T05:39:05.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:05 smithi187 conmon[36358]: cluster 2021-11-20T05:39:04.309552+0000 mgr.smithi058.zagdlk (mgr.14164) 341 : cluster 2021-11-20T05:39:05.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:05 smithi187 conmon[36358]: [DBG] pgmap v319: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 0 op/s 2021-11-20T05:39:05.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:05 smithi187 conmon[36358]: audit 2021-11-20T05:39:04.485017+0000 mon.smithi058 (mon.0) 900 : audit [DBG] from='client.? 172.21.15.58:0/1952185178' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:05.714 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:05 smithi058 conmon[33537]: cluster 2021-11-20T05:39:04.309552+0000 mgr.smithi058.zagdlk (mgr.14164) 341 : cluster 2021-11-20T05:39:05.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:05 smithi058 conmon[33537]: [DBG] pgmap v319: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 0 op/s 2021-11-20T05:39:05.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:05 smithi058 conmon[33537]: audit 2021-11-20T05:39:04.485017+0000 mon.smithi058 (mon.0) 900 2021-11-20T05:39:05.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:05 smithi058 conmon[33537]: : audit [DBG] from='client.? 172.21.15.58:0/1952185178' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:05.830 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.834+0000 7fce7b59e700 1 -- 172.21.15.58:0/1819818021 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce7c027a00 con 0x7fce7c0f7400 2021-11-20T05:39:05.831 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.835+0000 7fce6affd700 1 -- 172.21.15.58:0/1819818021 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2200 (secure 0 0 0) 0x7fce7004c000 con 0x7fce7c0f7400 2021-11-20T05:39:05.831 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:39:05.831 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:52.287814+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:39:05.833 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.836+0000 7fce7b59e700 1 -- 172.21.15.58:0/1819818021 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fce6405c300 msgr2=0x7fce6405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:05.833 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.836+0000 7fce7b59e700 1 --2- 172.21.15.58:0/1819818021 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fce6405c300 0x7fce6405e7c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fce7c0fd4d0 tx=0x7fce6c009b30).stop 2021-11-20T05:39:05.833 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.836+0000 7fce7b59e700 1 -- 172.21.15.58:0/1819818021 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce7c0f7400 msgr2=0x7fce7c0fbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:05.834 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.836+0000 7fce7b59e700 1 --2- 172.21.15.58:0/1819818021 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce7c0f7400 0x7fce7c0fbbc0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7fce70002700 tx=0x7fce70009d60).stop 2021-11-20T05:39:05.834 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.836+0000 7fce7b59e700 1 -- 172.21.15.58:0/1819818021 shutdown_connections 2021-11-20T05:39:05.834 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.836+0000 7fce7b59e700 1 --2- 172.21.15.58:0/1819818021 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fce6405c300 0x7fce6405e7c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:05.834 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.836+0000 7fce7b59e700 1 --2- 172.21.15.58:0/1819818021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce7c0f6a60 0x7fce7c0fd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:05.834 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.836+0000 7fce7b59e700 1 --2- 172.21.15.58:0/1819818021 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fce7c0f7400 0x7fce7c0fbbc0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:05.835 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.836+0000 7fce7b59e700 1 -- 172.21.15.58:0/1819818021 >> 172.21.15.58:0/1819818021 conn(0x7fce7c0f0360 msgr2=0x7fce7c0fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:05.835 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.836+0000 7fce7b59e700 1 -- 172.21.15.58:0/1819818021 shutdown_connections 2021-11-20T05:39:05.835 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:05.836+0000 7fce7b59e700 1 -- 172.21.15.58:0/1819818021 wait complete. 2021-11-20T05:39:05.835 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 100 2021-11-20T05:39:05.844 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:52.287814+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:39:05.845 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:rejoin 2021-11-20T05:39:06.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:06 smithi187 conmon[36358]: audit 2021-11-20T05:39:05.836642+0000 mon.smithi058 (mon.0) 901 : audit [DBG] 2021-11-20T05:39:06.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:06 smithi187 conmon[36358]: from='client.? 172.21.15.58:0/1819818021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:06.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:06 smithi058 conmon[33537]: audit 2021- 2021-11-20T05:39:06.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:06 smithi058 conmon[33537]: 11-20T05:39:05.836642+0000 mon.smithi058 (mon.0) 901 : audit [DBG] from='client.? 172.21.15.58:0/1819818021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:06.845 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:39:06.979 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.981+0000 7f9cb72b8700 1 -- 172.21.15.58:0/1210559136 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9cb00f5580 msgr2=0x7f9cb00f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:06.979 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.981+0000 7f9cb72b8700 1 --2- 172.21.15.58:0/1210559136 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9cb00f5580 0x7f9cb00f59a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f9cac004660 tx=0x7f9cac009b30).stop 2021-11-20T05:39:06.980 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.981+0000 7f9cb72b8700 1 -- 172.21.15.58:0/1210559136 shutdown_connections 2021-11-20T05:39:06.980 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.981+0000 7f9cb72b8700 1 --2- 172.21.15.58:0/1210559136 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb00f6820 0x7f9cb00f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:06.980 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.981+0000 7f9cb72b8700 1 --2- 172.21.15.58:0/1210559136 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9cb00f5580 0x7f9cb00f59a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:06.980 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.981+0000 7f9cb72b8700 1 -- 172.21.15.58:0/1210559136 >> 172.21.15.58:0/1210559136 conn(0x7f9cb00f0ce0 msgr2=0x7f9cb00f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:06.981 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.981+0000 7f9cb72b8700 1 -- 172.21.15.58:0/1210559136 shutdown_connections 2021-11-20T05:39:06.981 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.981+0000 7f9cb72b8700 1 -- 172.21.15.58:0/1210559136 wait complete. 2021-11-20T05:39:06.981 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.982+0000 7f9cb72b8700 1 Processor -- start 2021-11-20T05:39:06.981 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.982+0000 7f9cb72b8700 1 -- start start 2021-11-20T05:39:06.981 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.982+0000 7f9cb72b8700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9cb00f5580 0x7f9cb0060550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:06.982 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.982+0000 7f9cb72b8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb00f6820 0x7f9cb010bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:06.982 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.982+0000 7f9cb72b8700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb010c1d0 con 0x7f9cb00f5580 2021-11-20T05:39:06.982 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.982+0000 7f9cb72b8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb010c310 con 0x7f9cb00f6820 2021-11-20T05:39:06.982 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.982+0000 7f9cb5054700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9cb00f5580 0x7f9cb0060550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:06.983 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.982+0000 7f9cb5054700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9cb00f5580 0x7f9cb0060550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42526/0 (socket says 172.21.15.58:42526) 2021-11-20T05:39:06.983 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.982+0000 7f9cb5054700 1 -- 172.21.15.58:0/1512223191 learned_addr learned my addr 172.21.15.58:0/1512223191 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:39:06.983 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.982+0000 7f9cb5054700 1 -- 172.21.15.58:0/1512223191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb00f6820 msgr2=0x7f9cb010bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:06.983 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.982+0000 7f9cb5054700 1 --2- 172.21.15.58:0/1512223191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb00f6820 0x7f9cb010bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:39:06.984 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.982+0000 7f9cb5054700 1 -- 172.21.15.58:0/1512223191 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cac005040 con 0x7f9cb00f5580 2021-11-20T05:39:06.984 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.982+0000 7f9cb5054700 1 --2- 172.21.15.58:0/1512223191 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9cb00f5580 0x7f9cb0060550 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f9cac009f50 tx=0x7f9cac009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:06.984 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.982+0000 7f9ca5ffb700 1 -- 172.21.15.58:0/1512223191 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cac018070 con 0x7f9cb00f5580 2021-11-20T05:39:06.984 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.983+0000 7f9ca5ffb700 1 -- 172.21.15.58:0/1512223191 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9cac013410 con 0x7f9cb00f5580 2021-11-20T05:39:06.984 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.983+0000 7f9cb72b8700 1 -- 172.21.15.58:0/1512223191 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb010c510 con 0x7f9cb00f5580 2021-11-20T05:39:06.985 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.983+0000 7f9ca5ffb700 1 -- 172.21.15.58:0/1512223191 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cac01c620 con 0x7f9cb00f5580 2021-11-20T05:39:06.985 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.983+0000 7f9cb72b8700 1 -- 172.21.15.58:0/1512223191 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb010ca40 con 0x7f9cb00f5580 2021-11-20T05:39:06.986 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.984+0000 7f9cb72b8700 1 -- 172.21.15.58:0/1512223191 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb0046330 con 0x7f9cb00f5580 2021-11-20T05:39:06.986 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.984+0000 7f9ca5ffb700 1 -- 172.21.15.58:0/1512223191 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9cac01d420 con 0x7f9cb00f5580 2021-11-20T05:39:06.986 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.984+0000 7f9ca5ffb700 1 --2- 172.21.15.58:0/1512223191 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9c9c05c2a0 0x7f9c9c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:06.987 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.984+0000 7f9ca5ffb700 1 -- 172.21.15.58:0/1512223191 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f9cac0782e0 con 0x7f9cb00f5580 2021-11-20T05:39:06.987 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.985+0000 7f9cb4853700 1 --2- 172.21.15.58:0/1512223191 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9c9c05c2a0 0x7f9c9c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:06.987 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.985+0000 7f9cb4853700 1 --2- 172.21.15.58:0/1512223191 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9c9c05c2a0 0x7f9c9c05e760 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f9ca00047d0 tx=0x7f9ca0007a40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:06.987 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:06.988+0000 7f9ca5ffb700 1 -- 172.21.15.58:0/1512223191 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9cac04aa40 con 0x7f9cb00f5580 2021-11-20T05:39:07.179 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:07.183+0000 7f9cb72b8700 1 -- 172.21.15.58:0/1512223191 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cb00fdd10 con 0x7f9cb00f5580 2021-11-20T05:39:07.179 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:07.183+0000 7f9ca5ffb700 1 -- 172.21.15.58:0/1512223191 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2200 (secure 0 0 0) 0x7f9cac04e060 con 0x7f9cb00f5580 2021-11-20T05:39:07.180 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:39:07.180 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:52.287814+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:39:07.181 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:07.185+0000 7f9cb72b8700 1 -- 172.21.15.58:0/1512223191 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9c9c05c2a0 msgr2=0x7f9c9c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:07.181 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:07.185+0000 7f9cb72b8700 1 --2- 172.21.15.58:0/1512223191 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9c9c05c2a0 0x7f9c9c05e760 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f9ca00047d0 tx=0x7f9ca0007a40).stop 2021-11-20T05:39:07.182 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:07.185+0000 7f9cb72b8700 1 -- 172.21.15.58:0/1512223191 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9cb00f5580 msgr2=0x7f9cb0060550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:07.182 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:07.185+0000 7f9cb72b8700 1 --2- 172.21.15.58:0/1512223191 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9cb00f5580 0x7f9cb0060550 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f9cac009f50 tx=0x7f9cac009b30).stop 2021-11-20T05:39:07.182 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:07.185+0000 7f9cb72b8700 1 -- 172.21.15.58:0/1512223191 shutdown_connections 2021-11-20T05:39:07.182 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:07.185+0000 7f9cb72b8700 1 --2- 172.21.15.58:0/1512223191 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f9c9c05c2a0 0x7f9c9c05e760 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:07.183 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:07.185+0000 7f9cb72b8700 1 --2- 172.21.15.58:0/1512223191 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f9cb00f5580 0x7f9cb0060550 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:07.183 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:07.185+0000 7f9cb72b8700 1 --2- 172.21.15.58:0/1512223191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9cb00f6820 0x7f9cb010bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:39:07.183 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:07.185+0000 7f9cb72b8700 1 -- 172.21.15.58:0/1512223191 >> 172.21.15.58:0/1512223191 conn(0x7f9cb00f0ce0 msgr2=0x7f9cb00fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:07.183 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:07.185+0000 7f9cb72b8700 1 -- 172.21.15.58:0/1512223191 shutdown_connections 2021-11-20T05:39:07.184 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:07.185+0000 7f9cb72b8700 1 -- 172.21.15.58:0/1512223191 wait complete. 2021-11-20T05:39:07.184 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 100 2021-11-20T05:39:07.193 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:52.287814+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:39:07.193 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:rejoin 2021-11-20T05:39:07.465 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:07 smithi058 conmon[33537]: cluster 2021-11-20T05:39:06.310123+0000 2021-11-20T05:39:07.465 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:07 smithi058 conmon[33537]: mgr.smithi058.zagdlk (mgr.14164) 342 : cluster [DBG] pgmap v320: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:39:07.465 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:07 smithi058 conmon[33537]: audit 2021-11-20T05:39:07.185200+0000 mon.smithi058 (mon.0 2021-11-20T05:39:07.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:07 smithi058 conmon[33537]: ) 902 : audit [DBG] from='client.? 172.21.15.58:0/1512223191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:07.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:07 smithi187 conmon[36358]: cluster 2021-11-20T05:39:06.310123+0000 mgr.smithi058.zagdlk (mgr.14164) 342 : cluster 2021-11-20T05:39:07.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:07 smithi187 conmon[36358]: [DBG] pgmap v320: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:39:07.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:07 smithi187 conmon[36358]: audit 2021-11-20T05:39:07.185200+0000 mon.smithi058 (mon.0) 902 : audit [DBG] from='client.? 172.21.15.58:0/1512223191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:08.194 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:39:08.327 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.331+0000 7f187d7b3700 1 -- 172.21.15.58:0/1090115476 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18780f9880 msgr2=0x7f18780fbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:08.327 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.331+0000 7f187d7b3700 1 --2- 172.21.15.58:0/1090115476 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18780f9880 0x7f18780fbc70 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f1860004660 tx=0x7f1860009b30).stop 2021-11-20T05:39:08.328 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.331+0000 7f187d7b3700 1 -- 172.21.15.58:0/1090115476 shutdown_connections 2021-11-20T05:39:08.328 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.331+0000 7f187d7b3700 1 --2- 172.21.15.58:0/1090115476 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18780f9880 0x7f18780fbc70 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:08.328 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.331+0000 7f187d7b3700 1 --2- 172.21.15.58:0/1090115476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18780600c0 0x7f18780f9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:08.328 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.331+0000 7f187d7b3700 1 -- 172.21.15.58:0/1090115476 >> 172.21.15.58:0/1090115476 conn(0x7f18780f0ce0 msgr2=0x7f18780f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:08.328 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.331+0000 7f187d7b3700 1 -- 172.21.15.58:0/1090115476 shutdown_connections 2021-11-20T05:39:08.329 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.331+0000 7f187d7b3700 1 -- 172.21.15.58:0/1090115476 wait complete. 2021-11-20T05:39:08.331 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.331+0000 7f187d7b3700 1 Processor -- start 2021-11-20T05:39:08.332 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.332+0000 7f187d7b3700 1 -- start start 2021-11-20T05:39:08.332 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.332+0000 7f187d7b3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18780600c0 0x7f1878106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:08.332 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.332+0000 7f187d7b3700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18780f9880 0x7f1878107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:08.332 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.332+0000 7f187d7b3700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1878107940 con 0x7f18780f9880 2021-11-20T05:39:08.333 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.332+0000 7f187d7b3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1878107a80 con 0x7f18780600c0 2021-11-20T05:39:08.333 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.332+0000 7f18767fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18780f9880 0x7f1878107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:08.333 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.332+0000 7f18767fc700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18780f9880 0x7f1878107390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42538/0 (socket says 172.21.15.58:42538) 2021-11-20T05:39:08.333 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.332+0000 7f18767fc700 1 -- 172.21.15.58:0/1920489269 learned_addr learned my addr 172.21.15.58:0/1920489269 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:39:08.334 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.332+0000 7f18767fc700 1 -- 172.21.15.58:0/1920489269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18780600c0 msgr2=0x7f1878106ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:08.334 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.332+0000 7f18767fc700 1 --2- 172.21.15.58:0/1920489269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18780600c0 0x7f1878106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:39:08.334 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.332+0000 7f18767fc700 1 -- 172.21.15.58:0/1920489269 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1860005040 con 0x7f18780f9880 2021-11-20T05:39:08.334 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.332+0000 7f18767fc700 1 --2- 172.21.15.58:0/1920489269 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18780f9880 0x7f1878107390 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f1860007a70 tx=0x7f1860009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:08.335 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.332+0000 7f186f7fe700 1 -- 172.21.15.58:0/1920489269 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1860018070 con 0x7f18780f9880 2021-11-20T05:39:08.335 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.332+0000 7f186f7fe700 1 -- 172.21.15.58:0/1920489269 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1860009700 con 0x7f18780f9880 2021-11-20T05:39:08.335 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.332+0000 7f187d7b3700 1 -- 172.21.15.58:0/1920489269 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f187810c4c0 con 0x7f18780f9880 2021-11-20T05:39:08.335 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.332+0000 7f186f7fe700 1 -- 172.21.15.58:0/1920489269 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f186001c620 con 0x7f18780f9880 2021-11-20T05:39:08.336 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.332+0000 7f187d7b3700 1 -- 172.21.15.58:0/1920489269 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f187810ca80 con 0x7f18780f9880 2021-11-20T05:39:08.336 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.333+0000 7f187d7b3700 1 -- 172.21.15.58:0/1920489269 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1878027a00 con 0x7f18780f9880 2021-11-20T05:39:08.336 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.334+0000 7f186f7fe700 1 -- 172.21.15.58:0/1920489269 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f186001d420 con 0x7f18780f9880 2021-11-20T05:39:08.336 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.334+0000 7f186f7fe700 1 --2- 172.21.15.58:0/1920489269 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f186405c2a0 0x7f186405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:08.337 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.334+0000 7f186f7fe700 1 -- 172.21.15.58:0/1920489269 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f1860078310 con 0x7f18780f9880 2021-11-20T05:39:08.337 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.334+0000 7f1876ffd700 1 --2- 172.21.15.58:0/1920489269 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f186405c2a0 0x7f186405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:08.337 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.335+0000 7f1876ffd700 1 --2- 172.21.15.58:0/1920489269 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f186405c2a0 0x7f186405e760 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f18700079c0 tx=0x7f1870004940).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:08.337 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.339+0000 7f186f7fe700 1 -- 172.21.15.58:0/1920489269 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f186004aa70 con 0x7f18780f9880 2021-11-20T05:39:08.532 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.536+0000 7f187d7b3700 1 -- 172.21.15.58:0/1920489269 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18780fee30 con 0x7f18780f9880 2021-11-20T05:39:08.532 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.536+0000 7f186f7fe700 1 -- 172.21.15.58:0/1920489269 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2200 (secure 0 0 0) 0x7f186004e090 con 0x7f18780f9880 2021-11-20T05:39:08.533 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:39:08.533 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:38:52.287814+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:39:08.534 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.538+0000 7f187d7b3700 1 -- 172.21.15.58:0/1920489269 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f186405c2a0 msgr2=0x7f186405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:08.535 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.538+0000 7f187d7b3700 1 --2- 172.21.15.58:0/1920489269 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f186405c2a0 0x7f186405e760 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f18700079c0 tx=0x7f1870004940).stop 2021-11-20T05:39:08.535 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.538+0000 7f187d7b3700 1 -- 172.21.15.58:0/1920489269 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18780f9880 msgr2=0x7f1878107390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:08.535 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.538+0000 7f187d7b3700 1 --2- 172.21.15.58:0/1920489269 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18780f9880 0x7f1878107390 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f1860007a70 tx=0x7f1860009b30).stop 2021-11-20T05:39:08.535 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.538+0000 7f187d7b3700 1 -- 172.21.15.58:0/1920489269 shutdown_connections 2021-11-20T05:39:08.535 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.538+0000 7f187d7b3700 1 --2- 172.21.15.58:0/1920489269 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f186405c2a0 0x7f186405e760 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:08.536 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.538+0000 7f187d7b3700 1 --2- 172.21.15.58:0/1920489269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18780600c0 0x7f1878106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:39:08.536 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.538+0000 7f187d7b3700 1 --2- 172.21.15.58:0/1920489269 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f18780f9880 0x7f1878107390 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:08.536 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.538+0000 7f187d7b3700 1 -- 172.21.15.58:0/1920489269 >> 172.21.15.58:0/1920489269 conn(0x7f18780f0ce0 msgr2=0x7f18780f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:08.536 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.538+0000 7f187d7b3700 1 -- 172.21.15.58:0/1920489269 shutdown_connections 2021-11-20T05:39:08.537 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:08.538+0000 7f187d7b3700 1 -- 172.21.15.58:0/1920489269 wait complete. 2021-11-20T05:39:08.537 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 100 2021-11-20T05:39:08.546 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:38:52.287814+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:39:08.546 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:rejoin 2021-11-20T05:39:09.548 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:39:09.689 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.692+0000 7efe5160f700 1 -- 172.21.15.58:0/4271574133 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efe4c0f5500 msgr2=0x7efe4c0f5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:09.690 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.692+0000 7efe5160f700 1 --2- 172.21.15.58:0/4271574133 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efe4c0f5500 0x7efe4c0f5920 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7efe3c004660 tx=0x7efe3c009b30).stop 2021-11-20T05:39:09.690 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.692+0000 7efe5160f700 1 -- 172.21.15.58:0/4271574133 shutdown_connections 2021-11-20T05:39:09.690 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.692+0000 7efe5160f700 1 --2- 172.21.15.58:0/4271574133 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe4c0f67a0 0x7efe4c0f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:09.691 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.692+0000 7efe5160f700 1 --2- 172.21.15.58:0/4271574133 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efe4c0f5500 0x7efe4c0f5920 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:09.691 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.692+0000 7efe5160f700 1 -- 172.21.15.58:0/4271574133 >> 172.21.15.58:0/4271574133 conn(0x7efe4c0f0c60 msgr2=0x7efe4c0f3080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:09.691 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.692+0000 7efe5160f700 1 -- 172.21.15.58:0/4271574133 shutdown_connections 2021-11-20T05:39:09.691 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.693+0000 7efe5160f700 1 -- 172.21.15.58:0/4271574133 wait complete. 2021-11-20T05:39:09.691 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.693+0000 7efe5160f700 1 Processor -- start 2021-11-20T05:39:09.692 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.693+0000 7efe5160f700 1 -- start start 2021-11-20T05:39:09.692 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.693+0000 7efe5160f700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efe4c0f5500 0x7efe4c105d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:09.692 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.693+0000 7efe5160f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe4c0f67a0 0x7efe4c106240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:09.692 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.693+0000 7efe5160f700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4c1067f0 con 0x7efe4c0f5500 2021-11-20T05:39:09.693 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.693+0000 7efe5160f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4c106ff0 con 0x7efe4c0f67a0 2021-11-20T05:39:09.693 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.694+0000 7efe4affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efe4c0f5500 0x7efe4c105d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:09.693 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.694+0000 7efe4affd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efe4c0f5500 0x7efe4c105d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42548/0 (socket says 172.21.15.58:42548) 2021-11-20T05:39:09.693 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.694+0000 7efe4affd700 1 -- 172.21.15.58:0/49420961 learned_addr learned my addr 172.21.15.58:0/49420961 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:39:09.694 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.694+0000 7efe4affd700 1 -- 172.21.15.58:0/49420961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe4c0f67a0 msgr2=0x7efe4c106240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:09.694 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.694+0000 7efe4affd700 1 --2- 172.21.15.58:0/49420961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe4c0f67a0 0x7efe4c106240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:39:09.694 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.694+0000 7efe4affd700 1 -- 172.21.15.58:0/49420961 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3c005040 con 0x7efe4c0f5500 2021-11-20T05:39:09.694 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.694+0000 7efe4affd700 1 --2- 172.21.15.58:0/49420961 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efe4c0f5500 0x7efe4c105d50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7efe3c009f50 tx=0x7efe3c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:09.695 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.694+0000 7efe337fe700 1 -- 172.21.15.58:0/49420961 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe3c018070 con 0x7efe4c0f5500 2021-11-20T05:39:09.695 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.694+0000 7efe337fe700 1 -- 172.21.15.58:0/49420961 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe3c013410 con 0x7efe4c0f5500 2021-11-20T05:39:09.695 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.694+0000 7efe5160f700 1 -- 172.21.15.58:0/49420961 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe4c107270 con 0x7efe4c0f5500 2021-11-20T05:39:09.695 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.694+0000 7efe337fe700 1 -- 172.21.15.58:0/49420961 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe3c01c620 con 0x7efe4c0f5500 2021-11-20T05:39:09.695 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.694+0000 7efe5160f700 1 -- 172.21.15.58:0/49420961 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe4c1077d0 con 0x7efe4c0f5500 2021-11-20T05:39:09.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:09 smithi187 conmon[36358]: cluster 2021-11-20T05:39:08.310554+0000 mgr.smithi058.zagdlk (mgr.14164) 343 : cluster 2021-11-20T05:39:09.696 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:09 smithi187 conmon[36358]: [DBG] pgmap v321: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:39:09.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:09 smithi187 conmon[36358]: audit 2021-11-20T05:39:08.538076+0000 mon.smithi058 (mon.0) 903 : audit [DBG] from='client.? 172.21.15.58:0/1920489269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:09.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:09 smithi187 conmon[36358]: cluster 2021-11-20T05:39:09.284969+0000 mon.smithi058 (mon.0) 904 : cluster [DBG] 2021-11-20T05:39:09.697 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:09 smithi187 conmon[36358]: fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin(laggy or crashed),1=cephfs.smithi058.nuxpgf=up:active} 2021-11-20T05:39:09.698 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.695+0000 7efe5160f700 1 -- 172.21.15.58:0/49420961 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe4c0f48f0 con 0x7efe4c0f5500 2021-11-20T05:39:09.698 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.695+0000 7efe337fe700 1 -- 172.21.15.58:0/49420961 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7efe3c01d420 con 0x7efe4c0f5500 2021-11-20T05:39:09.699 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.696+0000 7efe337fe700 1 --2- 172.21.15.58:0/49420961 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efe3405c300 0x7efe3405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:09.699 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.696+0000 7efe337fe700 1 -- 172.21.15.58:0/49420961 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7efe3c0782e0 con 0x7efe4c0f5500 2021-11-20T05:39:09.699 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.696+0000 7efe4a7fc700 1 --2- 172.21.15.58:0/49420961 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efe3405c300 0x7efe3405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:09.699 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.696+0000 7efe4a7fc700 1 --2- 172.21.15.58:0/49420961 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efe3405c300 0x7efe3405e7c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7efe400047d0 tx=0x7efe40007a40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:09.700 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.700+0000 7efe337fe700 1 -- 172.21.15.58:0/49420961 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe3c04aa40 con 0x7efe4c0f5500 2021-11-20T05:39:09.714 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:09 smithi058 conmon[33537]: cluster 2021-11-20T05:39:08.310554+0000 mgr.smithi058.zagdlk (mgr.14164) 343 : cluster [DBG] pgmap v321: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:39:09.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:09 smithi058 conmon[33537]: audit 2021-11-20T05:39:08.538076+0000 mon.smithi058 (mon.0) 903 : audit [DBG] from='client.? 172.21.15.58:0/1920489269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:09.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:09 smithi058 conmon[33537]: cluster 2021-11-20T05:39:09.284969+0000 2021-11-20T05:39:09.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:09 smithi058 conmon[33537]: mon.smithi058 (mon.0) 904 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi187.wxduhu=up:rejoin(laggy or crashed),1=cephfs.smithi058.nuxpgf=up:active} 2021-11-20T05:39:09.891 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.896+0000 7efe5160f700 1 -- 172.21.15.58:0/49420961 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe4c045ad0 con 0x7efe4c0f5500 2021-11-20T05:39:09.891 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.896+0000 7efe337fe700 1 -- 172.21.15.58:0/49420961 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2248 (secure 0 0 0) 0x7efe3c04e060 con 0x7efe4c0f5500 2021-11-20T05:39:09.892 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:39:09.892 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:39:09.281224+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"laggy_since":"2021-11-20T05:39:09.281181+0000","export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:39:09.893 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.898+0000 7efe5160f700 1 -- 172.21.15.58:0/49420961 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efe3405c300 msgr2=0x7efe3405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:09.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.898+0000 7efe5160f700 1 --2- 172.21.15.58:0/49420961 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efe3405c300 0x7efe3405e7c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7efe400047d0 tx=0x7efe40007a40).stop 2021-11-20T05:39:09.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.898+0000 7efe5160f700 1 -- 172.21.15.58:0/49420961 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efe4c0f5500 msgr2=0x7efe4c105d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:09.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.898+0000 7efe5160f700 1 --2- 172.21.15.58:0/49420961 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efe4c0f5500 0x7efe4c105d50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7efe3c009f50 tx=0x7efe3c009b30).stop 2021-11-20T05:39:09.894 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.898+0000 7efe5160f700 1 -- 172.21.15.58:0/49420961 shutdown_connections 2021-11-20T05:39:09.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.898+0000 7efe5160f700 1 --2- 172.21.15.58:0/49420961 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7efe3405c300 0x7efe3405e7c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:09.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.898+0000 7efe5160f700 1 --2- 172.21.15.58:0/49420961 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7efe4c0f5500 0x7efe4c105d50 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:09.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.898+0000 7efe5160f700 1 --2- 172.21.15.58:0/49420961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe4c0f67a0 0x7efe4c106240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:39:09.895 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.898+0000 7efe5160f700 1 -- 172.21.15.58:0/49420961 >> 172.21.15.58:0/49420961 conn(0x7efe4c0f0c60 msgr2=0x7efe4c0fa1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:09.896 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.898+0000 7efe5160f700 1 -- 172.21.15.58:0/49420961 shutdown_connections 2021-11-20T05:39:09.896 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:09.898+0000 7efe5160f700 1 -- 172.21.15.58:0/49420961 wait complete. 2021-11-20T05:39:09.896 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 101 2021-11-20T05:39:09.905 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:39:09.281224+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'laggy_since': '2021-11-20T05:39:09.281181+0000', 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:39:09.905 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:rejoin 2021-11-20T05:39:10.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:10 smithi187 conmon[36358]: audit 2021-11-20T05:39:09.897539+0000 mon.smithi058 (mon.0) 905 : audit 2021-11-20T05:39:10.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:10 smithi187 conmon[36358]: [DBG] from='client.? 172.21.15.58:0/49420961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:10.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:10 smithi058 conmon[33537]: audit 2021-11-20T05:39:09.897539+0000 mon.smithi058 (mon.0) 905 : audit [DBG] from='client.? 172.21.15.58:0/49420961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:10.907 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:39:11.040 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.043+0000 7f0221f60700 1 -- 172.21.15.58:0/3180344884 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0f7920 msgr2=0x7f021c0f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:11.041 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.043+0000 7f0221f60700 1 --2- 172.21.15.58:0/3180344884 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0f7920 0x7f021c0f7d80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f0210004660 tx=0x7f0210009b30).stop 2021-11-20T05:39:11.041 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.043+0000 7f0221f60700 1 -- 172.21.15.58:0/3180344884 shutdown_connections 2021-11-20T05:39:11.041 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.043+0000 7f0221f60700 1 --2- 172.21.15.58:0/3180344884 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0f7920 0x7f021c0f7d80 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:11.041 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.043+0000 7f0221f60700 1 --2- 172.21.15.58:0/3180344884 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f021c0f6680 0x7f021c0f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:11.042 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.043+0000 7f0221f60700 1 -- 172.21.15.58:0/3180344884 >> 172.21.15.58:0/3180344884 conn(0x7f021c0f1de0 msgr2=0x7f021c0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:11.042 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.043+0000 7f0221f60700 1 -- 172.21.15.58:0/3180344884 shutdown_connections 2021-11-20T05:39:11.042 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.043+0000 7f0221f60700 1 -- 172.21.15.58:0/3180344884 wait complete. 2021-11-20T05:39:11.042 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.043+0000 7f0221f60700 1 Processor -- start 2021-11-20T05:39:11.043 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.044+0000 7f0221f60700 1 -- start start 2021-11-20T05:39:11.043 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.044+0000 7f0221f60700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0f6680 0x7f021c06b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:11.043 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.044+0000 7f0221f60700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f021c0f7920 0x7f021c069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:11.043 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.044+0000 7f0221f60700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c06bd70 con 0x7f021c0f6680 2021-11-20T05:39:11.044 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.044+0000 7f0221f60700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c06a310 con 0x7f021c0f7920 2021-11-20T05:39:11.044 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.044+0000 7f021b7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0f6680 0x7f021c06b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:11.044 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.044+0000 7f021b7fe700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0f6680 0x7f021c06b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42558/0 (socket says 172.21.15.58:42558) 2021-11-20T05:39:11.044 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.044+0000 7f021b7fe700 1 -- 172.21.15.58:0/3075337775 learned_addr learned my addr 172.21.15.58:0/3075337775 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:39:11.045 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.044+0000 7f021affd700 1 --2- 172.21.15.58:0/3075337775 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f021c0f7920 0x7f021c069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:11.045 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.044+0000 7f021b7fe700 1 -- 172.21.15.58:0/3075337775 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f021c0f7920 msgr2=0x7f021c069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:11.045 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.044+0000 7f021b7fe700 1 --2- 172.21.15.58:0/3075337775 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f021c0f7920 0x7f021c069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:11.045 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.044+0000 7f021b7fe700 1 -- 172.21.15.58:0/3075337775 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0210005040 con 0x7f021c0f6680 2021-11-20T05:39:11.046 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.044+0000 7f021b7fe700 1 --2- 172.21.15.58:0/3075337775 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0f6680 0x7f021c06b7c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f020c00c530 tx=0x7f020c007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:11.046 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.045+0000 7f0203fff700 1 -- 172.21.15.58:0/3075337775 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f020c00f650 con 0x7f021c0f6680 2021-11-20T05:39:11.046 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.045+0000 7f0203fff700 1 -- 172.21.15.58:0/3075337775 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f020c00fc60 con 0x7f021c0f6680 2021-11-20T05:39:11.046 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.045+0000 7f0221f60700 1 -- 172.21.15.58:0/3075337775 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021c06a590 con 0x7f021c0f6680 2021-11-20T05:39:11.047 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.045+0000 7f0203fff700 1 -- 172.21.15.58:0/3075337775 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f020c012d20 con 0x7f021c0f6680 2021-11-20T05:39:11.047 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.045+0000 7f0221f60700 1 -- 172.21.15.58:0/3075337775 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f021c06ab50 con 0x7f021c0f6680 2021-11-20T05:39:11.047 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.046+0000 7f0221f60700 1 -- 172.21.15.58:0/3075337775 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f021c027a00 con 0x7f021c0f6680 2021-11-20T05:39:11.048 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.046+0000 7f0203fff700 1 -- 172.21.15.58:0/3075337775 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f020c019030 con 0x7f021c0f6680 2021-11-20T05:39:11.048 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.046+0000 7f0203fff700 1 --2- 172.21.15.58:0/3075337775 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f020405c2a0 0x7f020405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:11.049 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.046+0000 7f0203fff700 1 -- 172.21.15.58:0/3075337775 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f020c0760d0 con 0x7f021c0f6680 2021-11-20T05:39:11.049 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.047+0000 7f021affd700 1 --2- 172.21.15.58:0/3075337775 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f020405c2a0 0x7f020405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:11.049 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.047+0000 7f021affd700 1 --2- 172.21.15.58:0/3075337775 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f020405c2a0 0x7f020405e760 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f0210013580 tx=0x7f0210009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:11.049 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.050+0000 7f0203fff700 1 -- 172.21.15.58:0/3075337775 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f020c048880 con 0x7f021c0f6680 2021-11-20T05:39:11.244 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.249+0000 7f0221f60700 1 -- 172.21.15.58:0/3075337775 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f021c0fee10 con 0x7f021c0f6680 2021-11-20T05:39:11.245 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.249+0000 7f0203fff700 1 -- 172.21.15.58:0/3075337775 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2248 (secure 0 0 0) 0x7f020c017020 con 0x7f021c0f6680 2021-11-20T05:39:11.246 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:39:11.246 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:39:09.281224+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"laggy_since":"2021-11-20T05:39:09.281181+0000","export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:39:11.248 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.251+0000 7f0221f60700 1 -- 172.21.15.58:0/3075337775 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f020405c2a0 msgr2=0x7f020405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:11.248 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.251+0000 7f0221f60700 1 --2- 172.21.15.58:0/3075337775 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f020405c2a0 0x7f020405e760 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f0210013580 tx=0x7f0210009b30).stop 2021-11-20T05:39:11.248 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.251+0000 7f0221f60700 1 -- 172.21.15.58:0/3075337775 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0f6680 msgr2=0x7f021c06b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:11.248 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.251+0000 7f0221f60700 1 --2- 172.21.15.58:0/3075337775 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0f6680 0x7f021c06b7c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f020c00c530 tx=0x7f020c007e10).stop 2021-11-20T05:39:11.249 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.251+0000 7f0221f60700 1 -- 172.21.15.58:0/3075337775 shutdown_connections 2021-11-20T05:39:11.249 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.251+0000 7f0221f60700 1 --2- 172.21.15.58:0/3075337775 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f020405c2a0 0x7f020405e760 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:11.249 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.251+0000 7f0221f60700 1 --2- 172.21.15.58:0/3075337775 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f021c0f6680 0x7f021c06b7c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:11.249 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.251+0000 7f0221f60700 1 --2- 172.21.15.58:0/3075337775 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f021c0f7920 0x7f021c069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:11.250 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.251+0000 7f0221f60700 1 -- 172.21.15.58:0/3075337775 >> 172.21.15.58:0/3075337775 conn(0x7f021c0f1de0 msgr2=0x7f021c0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:11.250 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.251+0000 7f0221f60700 1 -- 172.21.15.58:0/3075337775 shutdown_connections 2021-11-20T05:39:11.250 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:11.251+0000 7f0221f60700 1 -- 172.21.15.58:0/3075337775 wait complete. 2021-11-20T05:39:11.250 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 101 2021-11-20T05:39:11.258 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:39:09.281224+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'laggy_since': '2021-11-20T05:39:09.281181+0000', 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:39:11.258 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:rejoin 2021-11-20T05:39:11.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:11 smithi187 conmon[36358]: cluster 2021-11-20T05:39:10.311268+0000 2021-11-20T05:39:11.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:11 smithi187 conmon[36358]: mgr.smithi058.zagdlk (mgr.14164) 344 : cluster [DBG] pgmap v322: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:39:11.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:11 smithi187 conmon[36358]: audit 2021-11-20T05:39:11.250255 2021-11-20T05:39:11.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:11 smithi187 conmon[36358]: +0000 mon.smithi058 (mon.0) 906 : audit [DBG] from='client.? 172.21.15.58:0/3075337775' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:11.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:11 smithi058 conmon[33537]: cluster 2021-11-20T05:39:10.311268+0000 mgr.smithi058.zagdlk 2021-11-20T05:39:11.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:11 smithi058 conmon[33537]: (mgr.14164) 344 : cluster [DBG] pgmap v322: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:39:11.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:11 smithi058 conmon[33537]: audit 2021-11-20T05:39:11.250255+0000 mon.smithi058 (mon.0) 906 : audit 2021-11-20T05:39:11.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:11 smithi058 conmon[33537]: [DBG] from='client.? 172.21.15.58:0/3075337775' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:12.260 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:39:12.398 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.399+0000 7fae7942a700 1 -- 172.21.15.58:0/1551299829 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae740f6820 msgr2=0x7fae740f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:12.399 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.399+0000 7fae7942a700 1 --2- 172.21.15.58:0/1551299829 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae740f6820 0x7fae740f6c80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7fae68004660 tx=0x7fae68009b30).stop 2021-11-20T05:39:12.399 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.399+0000 7fae7942a700 1 -- 172.21.15.58:0/1551299829 shutdown_connections 2021-11-20T05:39:12.399 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.399+0000 7fae7942a700 1 --2- 172.21.15.58:0/1551299829 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae740f6820 0x7fae740f6c80 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:12.399 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.399+0000 7fae7942a700 1 --2- 172.21.15.58:0/1551299829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae740f5580 0x7fae740f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:12.399 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.399+0000 7fae7942a700 1 -- 172.21.15.58:0/1551299829 >> 172.21.15.58:0/1551299829 conn(0x7fae740f0ce0 msgr2=0x7fae740f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:12.400 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.399+0000 7fae7942a700 1 -- 172.21.15.58:0/1551299829 shutdown_connections 2021-11-20T05:39:12.400 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.399+0000 7fae7942a700 1 -- 172.21.15.58:0/1551299829 wait complete. 2021-11-20T05:39:12.400 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.400+0000 7fae7942a700 1 Processor -- start 2021-11-20T05:39:12.400 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.400+0000 7fae7942a700 1 -- start start 2021-11-20T05:39:12.401 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.400+0000 7fae7942a700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae740f5580 0x7fae74106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:12.401 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.400+0000 7fae7942a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae740f6820 0x7fae741073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:12.401 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.400+0000 7fae7942a700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae74107960 con 0x7fae740f5580 2021-11-20T05:39:12.401 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.400+0000 7fae7942a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae74107aa0 con 0x7fae740f6820 2021-11-20T05:39:12.401 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.400+0000 7fae72ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae740f5580 0x7fae74106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:12.402 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.400+0000 7fae72ffd700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae740f5580 0x7fae74106ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42568/0 (socket says 172.21.15.58:42568) 2021-11-20T05:39:12.402 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.400+0000 7fae72ffd700 1 -- 172.21.15.58:0/1291570350 learned_addr learned my addr 172.21.15.58:0/1291570350 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:39:12.402 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.401+0000 7fae727fc700 1 --2- 172.21.15.58:0/1291570350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae740f6820 0x7fae741073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:12.402 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.401+0000 7fae72ffd700 1 -- 172.21.15.58:0/1291570350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae740f6820 msgr2=0x7fae741073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:12.403 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.401+0000 7fae72ffd700 1 --2- 172.21.15.58:0/1291570350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae740f6820 0x7fae741073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:12.403 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.401+0000 7fae72ffd700 1 -- 172.21.15.58:0/1291570350 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae68005040 con 0x7fae740f5580 2021-11-20T05:39:12.403 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.401+0000 7fae72ffd700 1 --2- 172.21.15.58:0/1291570350 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae740f5580 0x7fae74106ec0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7fae64002700 tx=0x7fae64009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:12.403 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.401+0000 7fae5b7fe700 1 -- 172.21.15.58:0/1291570350 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae640079c0 con 0x7fae740f5580 2021-11-20T05:39:12.404 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.401+0000 7fae5b7fe700 1 -- 172.21.15.58:0/1291570350 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae6400f040 con 0x7fae740f5580 2021-11-20T05:39:12.404 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.401+0000 7fae7942a700 1 -- 172.21.15.58:0/1291570350 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7410c540 con 0x7fae740f5580 2021-11-20T05:39:12.404 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.401+0000 7fae5b7fe700 1 -- 172.21.15.58:0/1291570350 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae64014860 con 0x7fae740f5580 2021-11-20T05:39:12.404 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.401+0000 7fae7942a700 1 -- 172.21.15.58:0/1291570350 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7410cb00 con 0x7fae740f5580 2021-11-20T05:39:12.404 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.402+0000 7fae7942a700 1 -- 172.21.15.58:0/1291570350 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae740fdd10 con 0x7fae740f5580 2021-11-20T05:39:12.405 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.402+0000 7fae5b7fe700 1 -- 172.21.15.58:0/1291570350 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fae6401b060 con 0x7fae740f5580 2021-11-20T05:39:12.405 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.403+0000 7fae5b7fe700 1 --2- 172.21.15.58:0/1291570350 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fae5c05c2a0 0x7fae5c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:12.405 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.403+0000 7fae5b7fe700 1 -- 172.21.15.58:0/1291570350 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fae640762e0 con 0x7fae740f5580 2021-11-20T05:39:12.405 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.403+0000 7fae727fc700 1 --2- 172.21.15.58:0/1291570350 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fae5c05c2a0 0x7fae5c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:12.405 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.403+0000 7fae727fc700 1 --2- 172.21.15.58:0/1291570350 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fae5c05c2a0 0x7fae5c05e760 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fae68009f50 tx=0x7fae68009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:12.406 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.407+0000 7fae5b7fe700 1 -- 172.21.15.58:0/1291570350 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae64048a40 con 0x7fae740f5580 2021-11-20T05:39:12.598 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.603+0000 7fae7942a700 1 -- 172.21.15.58:0/1291570350 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae7410d200 con 0x7fae740f5580 2021-11-20T05:39:12.599 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.604+0000 7fae5b7fe700 1 -- 172.21.15.58:0/1291570350 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2248 (secure 0 0 0) 0x7fae6404c060 con 0x7fae740f5580 2021-11-20T05:39:12.600 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:39:12.600 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:39:09.281224+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"laggy_since":"2021-11-20T05:39:09.281181+0000","export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:39:12.601 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.606+0000 7fae7942a700 1 -- 172.21.15.58:0/1291570350 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fae5c05c2a0 msgr2=0x7fae5c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:12.601 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.606+0000 7fae7942a700 1 --2- 172.21.15.58:0/1291570350 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fae5c05c2a0 0x7fae5c05e760 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fae68009f50 tx=0x7fae68009b30).stop 2021-11-20T05:39:12.601 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.606+0000 7fae7942a700 1 -- 172.21.15.58:0/1291570350 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae740f5580 msgr2=0x7fae74106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:12.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.606+0000 7fae7942a700 1 --2- 172.21.15.58:0/1291570350 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae740f5580 0x7fae74106ec0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7fae64002700 tx=0x7fae64009d80).stop 2021-11-20T05:39:12.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.606+0000 7fae7942a700 1 -- 172.21.15.58:0/1291570350 shutdown_connections 2021-11-20T05:39:12.602 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.606+0000 7fae7942a700 1 --2- 172.21.15.58:0/1291570350 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7fae5c05c2a0 0x7fae5c05e760 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:12.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.606+0000 7fae7942a700 1 --2- 172.21.15.58:0/1291570350 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7fae740f5580 0x7fae74106ec0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:12.603 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.606+0000 7fae7942a700 1 --2- 172.21.15.58:0/1291570350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae740f6820 0x7fae741073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:12.604 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.606+0000 7fae7942a700 1 -- 172.21.15.58:0/1291570350 >> 172.21.15.58:0/1291570350 conn(0x7fae740f0ce0 msgr2=0x7fae74060260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:12.604 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.606+0000 7fae7942a700 1 -- 172.21.15.58:0/1291570350 shutdown_connections 2021-11-20T05:39:12.604 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:12.606+0000 7fae7942a700 1 -- 172.21.15.58:0/1291570350 wait complete. 2021-11-20T05:39:12.605 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 101 2021-11-20T05:39:12.613 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:39:09.281224+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'laggy_since': '2021-11-20T05:39:09.281181+0000', 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:39:12.613 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:rejoin 2021-11-20T05:39:13.615 DEBUG:teuthology.orchestra.run.smithi058:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T05:39:13.643 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:13 smithi058 conmon[33537]: cluster 2021-11-20T05:39:12.311721+0000 mgr.smithi058.zagdlk (mgr.14164 2021-11-20T05:39:13.644 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:13 smithi058 conmon[33537]: ) 345 : cluster [DBG] pgmap v323: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:39:13.644 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:13 smithi058 conmon[33537]: audit 2021-11-20T05:39:12.605038+0000 mon.smithi058 (mon.0) 907 : audit [DBG] from='client.? 172.21.15.58:0/1291570350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:13.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:13 smithi187 conmon[36358]: cluster 2021-11-20T05:39:12.311721+0000 mgr.smithi058.zagdlk (mgr.14164) 345 : cluster 2021-11-20T05:39:13.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:13 smithi187 conmon[36358]: [DBG] pgmap v323: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:39:13.693 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:13 smithi187 conmon[36358]: audit 2021-11-20T05:39:12.605038+0000 mon.smithi058 (mon.0) 907 : audit [DBG] from='client.? 172.21.15.58:0/1291570350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:13.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.752+0000 7f3afb0e2700 1 -- 172.21.15.58:0/3199261648 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3af40f5580 msgr2=0x7f3af40f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:13.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.752+0000 7f3afb0e2700 1 --2- 172.21.15.58:0/3199261648 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3af40f5580 0x7f3af40f59a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f3ae8004660 tx=0x7f3ae8009b30).stop 2021-11-20T05:39:13.749 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.752+0000 7f3afb0e2700 1 -- 172.21.15.58:0/3199261648 shutdown_connections 2021-11-20T05:39:13.750 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.752+0000 7f3afb0e2700 1 --2- 172.21.15.58:0/3199261648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3af40f6820 0x7f3af40f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:13.750 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.752+0000 7f3afb0e2700 1 --2- 172.21.15.58:0/3199261648 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3af40f5580 0x7f3af40f59a0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:13.750 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.752+0000 7f3afb0e2700 1 -- 172.21.15.58:0/3199261648 >> 172.21.15.58:0/3199261648 conn(0x7f3af40f0ce0 msgr2=0x7f3af40f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:13.750 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.752+0000 7f3afb0e2700 1 -- 172.21.15.58:0/3199261648 shutdown_connections 2021-11-20T05:39:13.751 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.752+0000 7f3afb0e2700 1 -- 172.21.15.58:0/3199261648 wait complete. 2021-11-20T05:39:13.751 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.752+0000 7f3afb0e2700 1 Processor -- start 2021-11-20T05:39:13.751 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.752+0000 7f3afb0e2700 1 -- start start 2021-11-20T05:39:13.751 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.753+0000 7f3afb0e2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3af40f5580 0x7f3af40f9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:13.751 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.753+0000 7f3afb0e2700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3af40f6820 0x7f3af40f7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:13.752 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.753+0000 7f3afb0e2700 1 -- --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af40f9ab0 con 0x7f3af40f6820 2021-11-20T05:39:13.752 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.753+0000 7f3afb0e2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af40f8050 con 0x7f3af40f5580 2021-11-20T05:39:13.752 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.753+0000 7f3af3fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3af40f6820 0x7f3af40f7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:13.752 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.753+0000 7f3af3fff700 1 --2- >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3af40f6820 0x7f3af40f7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.58:3300/0 says I am v2:172.21.15.58:42580/0 (socket says 172.21.15.58:42580) 2021-11-20T05:39:13.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.753+0000 7f3af3fff700 1 -- 172.21.15.58:0/1506515312 learned_addr learned my addr 172.21.15.58:0/1506515312 (peer_addr_for_me v2:172.21.15.58:0/0) 2021-11-20T05:39:13.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.753+0000 7f3af8e7e700 1 --2- 172.21.15.58:0/1506515312 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3af40f5580 0x7f3af40f9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:13.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.753+0000 7f3af3fff700 1 -- 172.21.15.58:0/1506515312 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3af40f5580 msgr2=0x7f3af40f9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:13.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.753+0000 7f3af3fff700 1 --2- 172.21.15.58:0/1506515312 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3af40f5580 0x7f3af40f9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:13.753 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.753+0000 7f3af3fff700 1 -- 172.21.15.58:0/1506515312 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae8005040 con 0x7f3af40f6820 2021-11-20T05:39:13.754 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.753+0000 7f3af3fff700 1 --2- 172.21.15.58:0/1506515312 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3af40f6820 0x7f3af40f7b60 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f3ae000c530 tx=0x7f3ae0007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:13.754 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.754+0000 7f3af17fa700 1 -- 172.21.15.58:0/1506515312 <== mon.0 v2:172.21.15.58:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ae000f650 con 0x7f3af40f6820 2021-11-20T05:39:13.754 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.754+0000 7f3af17fa700 1 -- 172.21.15.58:0/1506515312 <== mon.0 v2:172.21.15.58:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ae000fc60 con 0x7f3af40f6820 2021-11-20T05:39:13.754 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.754+0000 7f3afb0e2700 1 -- 172.21.15.58:0/1506515312 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af40f82d0 con 0x7f3af40f6820 2021-11-20T05:39:13.755 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.754+0000 7f3af17fa700 1 -- 172.21.15.58:0/1506515312 <== mon.0 v2:172.21.15.58:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ae0012c80 con 0x7f3af40f6820 2021-11-20T05:39:13.755 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.754+0000 7f3afb0e2700 1 -- 172.21.15.58:0/1506515312 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af40f8890 con 0x7f3af40f6820 2021-11-20T05:39:13.756 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.755+0000 7f3afb0e2700 1 -- 172.21.15.58:0/1506515312 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af4046330 con 0x7f3af40f6820 2021-11-20T05:39:13.756 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.755+0000 7f3af17fa700 1 -- 172.21.15.58:0/1506515312 <== mon.0 v2:172.21.15.58:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3ae0010070 con 0x7f3af40f6820 2021-11-20T05:39:13.757 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.755+0000 7f3af17fa700 1 --2- 172.21.15.58:0/1506515312 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3ae405c2a0 0x7f3ae405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:39:13.757 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.756+0000 7f3af17fa700 1 -- 172.21.15.58:0/1506515312 <== mon.0 v2:172.21.15.58:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f3ae0076160 con 0x7f3af40f6820 2021-11-20T05:39:13.757 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.756+0000 7f3af8e7e700 1 --2- 172.21.15.58:0/1506515312 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3ae405c2a0 0x7f3ae405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:39:13.757 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.756+0000 7f3af8e7e700 1 --2- 172.21.15.58:0/1506515312 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3ae405c2a0 0x7f3ae405e760 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f3ae8004e30 tx=0x7f3ae8009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:39:13.758 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.759+0000 7f3af17fa700 1 -- 172.21.15.58:0/1506515312 <== mon.0 v2:172.21.15.58:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ae00452b0 con 0x7f3af40f6820 2021-11-20T05:39:13.950 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.955+0000 7f3afb0e2700 1 -- 172.21.15.58:0/1506515312 --> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3af40fdd10 con 0x7f3af40f6820 2021-11-20T05:39:13.951 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.956+0000 7f3af17fa700 1 -- 172.21.15.58:0/1506515312 <== mon.0 v2:172.21.15.58:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+2248 (secure 0 0 0) 0x7f3ae004c2f0 con 0x7f3af40f6820 2021-11-20T05:39:13.952 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-11-20T05:39:13.952 INFO:teuthology.orchestra.run.smithi058.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-11-20T05:34:06.306214+0000","modified":"2021-11-20T05:39:09.281224+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24617,"mds_1":14444},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi058.nuxpgf","rank":1,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.58:6829/4151830882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.58:6828","nonce":4151830882},{"type":"v1","addr":"172.21.15.58:6829","nonce":4151830882}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24617":{"gid":24617,"name":"cephfs.smithi187.wxduhu","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.187:6825/1414418853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":1414418853},{"type":"v1","addr":"172.21.15.187:6825","nonce":1414418853}]},"join_fscid":1,"laggy_since":"2021-11-20T05:39:09.281181+0000","export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T05:39:13.953 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.957+0000 7f3afb0e2700 1 -- 172.21.15.58:0/1506515312 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3ae405c2a0 msgr2=0x7f3ae405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:13.953 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.957+0000 7f3afb0e2700 1 --2- 172.21.15.58:0/1506515312 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3ae405c2a0 0x7f3ae405e760 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f3ae8004e30 tx=0x7f3ae8009b30).stop 2021-11-20T05:39:13.954 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.957+0000 7f3afb0e2700 1 -- 172.21.15.58:0/1506515312 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3af40f6820 msgr2=0x7f3af40f7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:39:13.954 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.957+0000 7f3afb0e2700 1 --2- 172.21.15.58:0/1506515312 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3af40f6820 0x7f3af40f7b60 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f3ae000c530 tx=0x7f3ae0007e20).stop 2021-11-20T05:39:13.954 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.957+0000 7f3afb0e2700 1 -- 172.21.15.58:0/1506515312 shutdown_connections 2021-11-20T05:39:13.954 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.957+0000 7f3afb0e2700 1 --2- 172.21.15.58:0/1506515312 >> [v2:172.21.15.58:6800/2097301011,v1:172.21.15.58:6801/2097301011] conn(0x7f3ae405c2a0 0x7f3ae405e760 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:13.955 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.957+0000 7f3afb0e2700 1 --2- 172.21.15.58:0/1506515312 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3af40f5580 0x7f3af40f9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:13.955 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.957+0000 7f3afb0e2700 1 --2- 172.21.15.58:0/1506515312 >> [v2:172.21.15.58:3300/0,v1:172.21.15.58:6789/0] conn(0x7f3af40f6820 0x7f3af40f7b60 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:39:13.955 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.957+0000 7f3afb0e2700 1 -- 172.21.15.58:0/1506515312 >> 172.21.15.58:0/1506515312 conn(0x7f3af40f0ce0 msgr2=0x7f3af4060260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:39:13.955 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.958+0000 7f3afb0e2700 1 -- 172.21.15.58:0/1506515312 shutdown_connections 2021-11-20T05:39:13.956 INFO:teuthology.orchestra.run.smithi058.stderr:2021-11-20T05:39:13.958+0000 7f3afb0e2700 1 -- 172.21.15.58:0/1506515312 wait complete. 2021-11-20T05:39:13.956 INFO:teuthology.orchestra.run.smithi058.stderr:dumped fsmap epoch 101 2021-11-20T05:39:13.964 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-11-20T05:34:06.306214+0000', 'modified': '2021-11-20T05:39:09.281224+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24617, 'mds_1': 14444}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14444': {'gid': 14444, 'name': 'cephfs.smithi058.nuxpgf', 'rank': 1, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.58:6829/4151830882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.58:6828', 'nonce': 4151830882}, {'type': 'v1', 'addr': '172.21.15.58:6829', 'nonce': 4151830882}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24617': {'gid': 24617, 'name': 'cephfs.smithi187.wxduhu', 'rank': 0, 'incarnation': 97, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.187:6825/1414418853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 1414418853}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 1414418853}]}, 'join_fscid': 1, 'laggy_since': '2021-11-20T05:39:09.281181+0000', 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T05:39:13.965 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24617:up:rejoin 2021-11-20T05:39:14.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:14 smithi187 conmon[36358]: audit 2021-11-20T05:39:13.956974+0000 mon.smithi058 (mon.0 2021-11-20T05:39:14.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:14 smithi187 conmon[36358]: ) 908 : audit [DBG] from='client.? 172.21.15.58:0/1506515312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:14.715 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:14 smithi058 conmon[33537]: audit 2021-11-20T05:39:13.956974+0000 mon.smithi058 (mon.0) 908 : audit [DBG] from='client.? 172.21.15.58:0/1506515312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T05:39:14.966 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 101, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2021-11-20T05:34:06.306214+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 101, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_14444": { "addr": "172.21.15.58:6829/4151830882", "addrs": { "addrvec": [ { "addr": "172.21.15.58:6828", "nonce": 4151830882, "type": "v2" }, { "addr": "172.21.15.58:6829", "nonce": 4151830882, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 14444, "incarnation": 7, "join_fscid": 1, "name": "cephfs.smithi058.nuxpgf", "rank": 1, "state": "up:active", "state_seq": 2 }, "gid_24617": { "addr": "172.21.15.187:6825/1414418853", "addrs": { "addrvec": [ { "addr": "172.21.15.187:6824", "nonce": 1414418853, "type": "v2" }, { "addr": "172.21.15.187:6825", "nonce": 1414418853, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 24617, "incarnation": 97, "join_fscid": 1, "laggy_since": "2021-11-20T05:39:09.281181+0000", "name": "cephfs.smithi187.wxduhu", "rank": 0, "state": "up:rejoin", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 2, "metadata_pool": 2, "modified": "2021-11-20T05:39:09.281224+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 1, "stopped": [], "tableserver": 0, "up": { "mds_0": 24617, "mds_1": 14444 } } } ], "standbys": [] } 2021-11-20T05:39:14.967 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/kclient.py", line 112, in task kernel_mount.mount(mntopts=client_config.get('mntopts', [])) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/kernel_mount.py", line 44, in mount self.setupfs(name=self.cephfs_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/mount.py", line 161, in setupfs self.fs.wait_for_daemons() File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/filesystem.py", line 1109, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy 2021-11-20T05:39:15.084 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4a1bc649f0bf4d2a807702083b707475 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/kclient.py", line 112, in task kernel_mount.mount(mntopts=client_config.get('mntopts', [])) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/kernel_mount.py", line 44, in mount self.setupfs(name=self.cephfs_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/mount.py", line 161, in setupfs self.fs.wait_for_daemons() File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/filesystem.py", line 1109, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy 2021-11-20T05:39:15.088 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2021-11-20T05:39:15.107 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2021-11-20T05:39:15.142 INFO:tasks.cephadm:Teardown begin 2021-11-20T05:39:15.143 DEBUG:teuthology.orchestra.run.smithi058:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-20T05:39:15.195 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-20T05:39:15.243 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2021-11-20T05:39:15.244 DEBUG:teuthology.orchestra.run.smithi058:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2021-11-20T05:39:15.263 DEBUG:teuthology.orchestra.run.smithi187:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2021-11-20T05:39:15.280 INFO:tasks.cephadm:Stopping all daemons... 2021-11-20T05:39:15.281 INFO:tasks.cephadm.mon.smithi058:Stopping mon.smithi058... 2021-11-20T05:39:15.281 DEBUG:teuthology.orchestra.run.smithi058:> sudo systemctl stop ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mon.smithi058 2021-11-20T05:39:15.465 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:15 smithi058 systemd[1]: Stopping Ceph mon.smithi058 for b33f8c4a-49c2-11ec-8c2c-001a4aab830c... 2021-11-20T05:39:15.465 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:15 smithi058 conmon[33537]: cluster 2021-11-20T05:39:14.312201+0000 mgr.smithi058.zagdlk (mgr.14164) 2021-11-20T05:39:15.466 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:15 smithi058 conmon[33537]: 346 : cluster [DBG] pgmap v324: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:39:15.692 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:15 smithi187 conmon[36358]: cluster 2021-11-20T05:39:14.312201+0000 mgr.smithi058.zagdlk (mgr.14164) 346 : cluster [DBG] pgmap v324: 65 pgs: 65 active+clean; 3.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T05:39:15.753 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:15 smithi058 conmon[33537]: debug 2021-11-20T05:39:15.502+0000 7ff1ac708700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi058 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-11-20T05:39:15.754 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:15 smithi058 conmon[33537]: debug 2021-11-20T05:39:15.502+0000 7ff1ac708700 -1 mon.smithi058@0(leader) e2 *** Got Signal Terminated *** 2021-11-20T05:39:16.064 INFO:journalctl@ceph.mon.smithi058.smithi058.stdout:Nov 20 05:39:15 smithi058 podman[95141]: ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c-mon.smithi058 2021-11-20T05:39:16.068 DEBUG:teuthology.orchestra.run.smithi058:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mon.smithi058.service' 2021-11-20T05:39:16.120 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T05:39:16.121 INFO:tasks.cephadm.mon.smithi058:Stopped mon.smithi058 2021-11-20T05:39:16.121 INFO:tasks.cephadm.mon.smithi187:Stopping mon.smithi187... 2021-11-20T05:39:16.122 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mon.smithi187 2021-11-20T05:39:16.442 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:16 smithi187 systemd[1]: Stopping Ceph mon.smithi187 for b33f8c4a-49c2-11ec-8c2c-001a4aab830c... 2021-11-20T05:39:16.443 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:16 smithi187 conmon[36358]: debug 2021-11-20T05:39:16.322+0000 7f4f548e4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi187 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-11-20T05:39:16.443 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:16 smithi187 conmon[36358]: debug 2021-11-20T05:39:16.322+0000 7f4f548e4700 -1 mon.smithi187@1(peon) e2 *** Got Signal Terminated *** 2021-11-20T05:39:17.085 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Nov 20 05:39:16 smithi187 podman[60332]: ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c-mon.smithi187 2021-11-20T05:39:17.089 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@mon.smithi187.service' 2021-11-20T05:39:17.144 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T05:39:17.145 INFO:tasks.cephadm.mon.smithi187:Stopped mon.smithi187 2021-11-20T05:39:17.146 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2021-11-20T05:39:17.146 DEBUG:teuthology.orchestra.run.smithi058:> sudo systemctl stop ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.0 2021-11-20T05:39:17.444 INFO:journalctl@ceph.osd.0.smithi058.stdout:Nov 20 05:39:17 smithi058 systemd[1]: Stopping Ceph osd.0 for b33f8c4a-49c2-11ec-8c2c-001a4aab830c... 2021-11-20T05:39:17.445 INFO:journalctl@ceph.osd.0.smithi058.stdout:Nov 20 05:39:17 smithi058 conmon[50697]: debug 2021-11-20T05:39:17.289+0000 7f7047fa0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T05:39:17.445 INFO:journalctl@ceph.osd.0.smithi058.stdout:Nov 20 05:39:17 smithi058 conmon[50697]: debug 2021-11-20T05:39:17.289+0000 7f7047fa0700 -1 osd.0 58 *** Got signal Terminated *** 2021-11-20T05:39:17.445 INFO:journalctl@ceph.osd.0.smithi058.stdout:Nov 20 05:39:17 smithi058 conmon[50697]: debug 2021-11-20T05:39:17.289+0000 7f7047fa0700 -1 osd.0 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T05:39:17.715 INFO:journalctl@ceph.osd.0.smithi058.stdout:Nov 20 05:39:17 smithi058 podman[95285]: ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c-osd.0 2021-11-20T05:39:18.198 INFO:journalctl@ceph.osd.0.smithi058.stdout:Nov 20 05:39:17 smithi058 podman[95448]: 2021-11-20T05:39:18.750 DEBUG:teuthology.orchestra.run.smithi058:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.0.service' 2021-11-20T05:39:18.837 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T05:39:18.838 INFO:tasks.cephadm.osd.0:Stopped osd.0 2021-11-20T05:39:18.838 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2021-11-20T05:39:18.839 DEBUG:teuthology.orchestra.run.smithi058:> sudo systemctl stop ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.1 2021-11-20T05:39:19.144 INFO:journalctl@ceph.osd.1.smithi058.stdout:Nov 20 05:39:18 smithi058 systemd[1]: Stopping Ceph osd.1 for b33f8c4a-49c2-11ec-8c2c-001a4aab830c... 2021-11-20T05:39:19.144 INFO:journalctl@ceph.osd.1.smithi058.stdout:Nov 20 05:39:18 smithi058 conmon[55656]: debug 2021-11-20T05:39:18.994+0000 7faf89270700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T05:39:19.145 INFO:journalctl@ceph.osd.1.smithi058.stdout:Nov 20 05:39:18 smithi058 conmon[55656]: debug 2021-11-20T05:39:18.994+0000 7faf89270700 -1 osd.1 58 *** Got signal Terminated *** 2021-11-20T05:39:19.145 INFO:journalctl@ceph.osd.1.smithi058.stdout:Nov 20 05:39:18 smithi058 conmon[55656]: debug 2021-11-20T05:39:18.994+0000 7faf89270700 -1 osd.1 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T05:39:19.464 INFO:journalctl@ceph.osd.1.smithi058.stdout:Nov 20 05:39:19 smithi058 podman[95611]: ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c-osd.1 2021-11-20T05:39:19.964 INFO:journalctl@ceph.osd.1.smithi058.stdout:Nov 20 05:39:19 smithi058 podman[95774]: 2021-11-20T05:39:20.469 DEBUG:teuthology.orchestra.run.smithi058:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.1.service' 2021-11-20T05:39:20.510 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T05:39:20.511 INFO:tasks.cephadm.osd.1:Stopped osd.1 2021-11-20T05:39:20.511 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2021-11-20T05:39:20.512 DEBUG:teuthology.orchestra.run.smithi058:> sudo systemctl stop ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.2 2021-11-20T05:39:20.834 INFO:journalctl@ceph.osd.2.smithi058.stdout:Nov 20 05:39:20 smithi058 systemd[1]: Stopping Ceph osd.2 for b33f8c4a-49c2-11ec-8c2c-001a4aab830c... 2021-11-20T05:39:20.835 INFO:journalctl@ceph.osd.2.smithi058.stdout:Nov 20 05:39:20 smithi058 conmon[61992]: debug 2021-11-20T05:39:20.677+0000 7f91b3b87700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T05:39:20.835 INFO:journalctl@ceph.osd.2.smithi058.stdout:Nov 20 05:39:20 smithi058 conmon[61992]: debug 2021-11-20T05:39:20.677+0000 7f91b3b87700 -1 osd.2 58 *** Got signal Terminated *** 2021-11-20T05:39:20.836 INFO:journalctl@ceph.osd.2.smithi058.stdout:Nov 20 05:39:20 smithi058 conmon[61992]: debug 2021-11-20T05:39:20.677+0000 7f91b3b87700 -1 osd.2 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T05:39:21.214 INFO:journalctl@ceph.osd.2.smithi058.stdout:Nov 20 05:39:20 smithi058 podman[95931]: ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c-osd.2 2021-11-20T05:39:21.522 INFO:journalctl@ceph.osd.2.smithi058.stdout:Nov 20 05:39:21 smithi058 podman[96097]: 2021-11-20T05:39:22.240 DEBUG:teuthology.orchestra.run.smithi058:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.2.service' 2021-11-20T05:39:22.280 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T05:39:22.281 INFO:tasks.cephadm.osd.2:Stopped osd.2 2021-11-20T05:39:22.281 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2021-11-20T05:39:22.282 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.3 2021-11-20T05:39:22.692 INFO:journalctl@ceph.osd.3.smithi187.stdout:Nov 20 05:39:22 smithi187 systemd[1]: Stopping Ceph osd.3 for b33f8c4a-49c2-11ec-8c2c-001a4aab830c... 2021-11-20T05:39:22.693 INFO:journalctl@ceph.osd.3.smithi187.stdout:Nov 20 05:39:22 smithi187 conmon[42957]: debug 2021-11-20T05:39:22.434+0000 7f68510e1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T05:39:22.694 INFO:journalctl@ceph.osd.3.smithi187.stdout:Nov 20 05:39:22 smithi187 conmon[42957]: debug 2021-11-20T05:39:22.434+0000 7f68510e1700 -1 osd.3 58 *** Got signal Terminated *** 2021-11-20T05:39:22.694 INFO:journalctl@ceph.osd.3.smithi187.stdout:Nov 20 05:39:22 smithi187 conmon[42957]: debug 2021-11-20T05:39:22.434+0000 7f68510e1700 -1 osd.3 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T05:39:23.001 INFO:journalctl@ceph.osd.3.smithi187.stdout:Nov 20 05:39:22 smithi187 podman[60465]: ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c-osd.3 2021-11-20T05:39:23.442 INFO:journalctl@ceph.osd.3.smithi187.stdout:Nov 20 05:39:23 smithi187 podman[60632]: 2021-11-20T05:39:23.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.3.service' 2021-11-20T05:39:24.020 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T05:39:24.020 INFO:tasks.cephadm.osd.3:Stopped osd.3 2021-11-20T05:39:24.021 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2021-11-20T05:39:24.021 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.4 2021-11-20T05:39:24.345 INFO:journalctl@ceph.osd.4.smithi187.stdout:Nov 20 05:39:24 smithi187 systemd[1]: Stopping Ceph osd.4 for b33f8c4a-49c2-11ec-8c2c-001a4aab830c... 2021-11-20T05:39:24.346 INFO:journalctl@ceph.osd.4.smithi187.stdout:Nov 20 05:39:24 smithi187 conmon[47832]: debug 2021-11-20T05:39:24.171+0000 7f8583888700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T05:39:24.346 INFO:journalctl@ceph.osd.4.smithi187.stdout:Nov 20 05:39:24 smithi187 conmon[47832]: debug 2021-11-20T05:39:24.171+0000 7f8583888700 -1 osd.4 58 *** Got signal Terminated *** 2021-11-20T05:39:24.346 INFO:journalctl@ceph.osd.4.smithi187.stdout:Nov 20 05:39:24 smithi187 conmon[47832]: debug 2021-11-20T05:39:24.171+0000 7f8583888700 -1 osd.4 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T05:39:24.692 INFO:journalctl@ceph.osd.4.smithi187.stdout:Nov 20 05:39:24 smithi187 podman[60793]: ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c-osd.4 2021-11-20T05:39:25.172 INFO:journalctl@ceph.osd.4.smithi187.stdout:Nov 20 05:39:24 smithi187 podman[60958]: 2021-11-20T05:39:25.542 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.4.service' 2021-11-20T05:39:25.579 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T05:39:25.580 INFO:tasks.cephadm.osd.4:Stopped osd.4 2021-11-20T05:39:25.581 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2021-11-20T05:39:25.582 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.5 2021-11-20T05:39:25.911 INFO:journalctl@ceph.osd.5.smithi187.stdout:Nov 20 05:39:25 smithi187 systemd[1]: Stopping Ceph osd.5 for b33f8c4a-49c2-11ec-8c2c-001a4aab830c... 2021-11-20T05:39:25.912 INFO:journalctl@ceph.osd.5.smithi187.stdout:Nov 20 05:39:25 smithi187 conmon[52734]: debug 2021-11-20T05:39:25.741+0000 7f11a9a09700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T05:39:25.912 INFO:journalctl@ceph.osd.5.smithi187.stdout:Nov 20 05:39:25 smithi187 conmon[52734]: debug 2021-11-20T05:39:25.741+0000 7f11a9a09700 -1 osd.5 58 *** Got signal Terminated *** 2021-11-20T05:39:25.912 INFO:journalctl@ceph.osd.5.smithi187.stdout:Nov 20 05:39:25 smithi187 conmon[52734]: debug 2021-11-20T05:39:25.741+0000 7f11a9a09700 -1 osd.5 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T05:39:26.192 INFO:journalctl@ceph.osd.5.smithi187.stdout:Nov 20 05:39:26 smithi187 podman[61121]: ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c-osd.5 2021-11-20T05:39:26.692 INFO:journalctl@ceph.osd.5.smithi187.stdout:Nov 20 05:39:26 smithi187 podman[61284]: 2021-11-20T05:39:27.102 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b33f8c4a-49c2-11ec-8c2c-001a4aab830c@osd.5.service' 2021-11-20T05:39:27.140 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T05:39:27.140 INFO:tasks.cephadm.osd.5:Stopped osd.5 2021-11-20T05:39:27.141 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c --force --keep-logs 2021-11-20T05:39:36.857 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c --force --keep-logs 2021-11-20T05:39:39.748 DEBUG:teuthology.orchestra.run.smithi058:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-20T05:39:39.785 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-20T05:39:39.814 INFO:tasks.cephadm:Archiving crash dumps... 2021-11-20T05:39:39.816 DEBUG:teuthology.misc:Transferring archived files from smithi058:/var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514741/remote/ubuntu@smithi058.front.sepia.ceph.com/crash 2021-11-20T05:39:39.818 DEBUG:teuthology.orchestra.run.smithi058:> sudo tar cz -f - -C /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/crash -- . 2021-11-20T05:39:39.871 INFO:teuthology.orchestra.run.smithi058.stderr:tar: /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/crash: Cannot open: No such file or directory 2021-11-20T05:39:39.872 INFO:teuthology.orchestra.run.smithi058.stderr:tar: Error is not recoverable: exiting now 2021-11-20T05:39:39.874 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514741/remote/ubuntu@smithi187.front.sepia.ceph.com/crash 2021-11-20T05:39:39.877 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar cz -f - -C /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/crash -- . 2021-11-20T05:39:39.919 INFO:teuthology.orchestra.run.smithi187.stderr:tar: /var/lib/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/crash: Cannot open: No such file or directory 2021-11-20T05:39:39.920 INFO:teuthology.orchestra.run.smithi187.stderr:tar: Error is not recoverable: exiting now 2021-11-20T05:39:39.921 INFO:tasks.cephadm:Checking cluster log for badness... 2021-11-20T05:39:39.922 DEBUG:teuthology.orchestra.run.smithi058:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2021-11-20T05:39:39.955 INFO:teuthology.orchestra.run.smithi058.stderr:grep: /var/log/ceph/b33f8c4a-49c2-11ec-8c2c-001a4aab830c/ceph.log: No such file or directory 2021-11-20T05:39:39.957 INFO:tasks.cephadm:Compressing logs... 2021-11-20T05:39:39.957 DEBUG:teuthology.orchestra.run.smithi058:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-20T05:39:40.000 DEBUG:teuthology.orchestra.run.smithi187:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-20T05:39:40.025 INFO:teuthology.orchestra.run.smithi187.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2021-11-20T05:39:40.026 INFO:teuthology.orchestra.run.smithi058.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2021-11-20T05:39:41.178 INFO:tasks.cephadm:Archiving logs... 2021-11-20T05:39:41.179 DEBUG:teuthology.misc:Transferring archived files from smithi058:/var/log/ceph to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514741/remote/ubuntu@smithi058.front.sepia.ceph.com/log 2021-11-20T05:39:41.180 DEBUG:teuthology.orchestra.run.smithi058:> sudo tar cz -f - -C /var/log/ceph -- . 2021-11-20T05:39:41.522 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/log/ceph to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514741/remote/ubuntu@smithi187.front.sepia.ceph.com/log 2021-11-20T05:39:41.524 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar cz -f - -C /var/log/ceph -- . 2021-11-20T05:39:41.697 INFO:tasks.cephadm:Removing cluster... 2021-11-20T05:39:41.697 DEBUG:teuthology.orchestra.run.smithi058:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c --force 2021-11-20T05:39:42.234 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b33f8c4a-49c2-11ec-8c2c-001a4aab830c --force 2021-11-20T05:39:42.748 INFO:tasks.cephadm:Removing cephadm ... 2021-11-20T05:39:42.749 DEBUG:teuthology.orchestra.run.smithi058:> rm -rf /home/ubuntu/cephtest/cephadm 2021-11-20T05:39:42.767 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf /home/ubuntu/cephtest/cephadm 2021-11-20T05:39:42.783 INFO:tasks.cephadm:Teardown complete 2021-11-20T05:39:42.784 DEBUG:teuthology.run_tasks:Unwinding manager install 2021-11-20T05:39:42.804 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2021-11-20T05:39:42.804 DEBUG:teuthology.orchestra.run.smithi058:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-11-20T05:39:42.809 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-11-20T05:39:42.898 DEBUG:teuthology.run_tasks:Unwinding manager clock 2021-11-20T05:39:42.911 INFO:teuthology.task.clock:Checking final clock skew... 2021-11-20T05:39:42.912 DEBUG:teuthology.orchestra.run.smithi058:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-20T05:39:42.915 DEBUG:teuthology.orchestra.run.smithi187:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-20T05:39:42.929 INFO:teuthology.orchestra.run.smithi058.stderr:bash: ntpq: command not found 2021-11-20T05:39:42.930 INFO:teuthology.orchestra.run.smithi187.stderr:bash: ntpq: command not found 2021-11-20T05:39:42.932 INFO:teuthology.orchestra.run.smithi058.stdout:210 Number of sources = 4 2021-11-20T05:39:42.932 INFO:teuthology.orchestra.run.smithi058.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-20T05:39:42.933 INFO:teuthology.orchestra.run.smithi058.stdout:=============================================================================== 2021-11-20T05:39:42.933 INFO:teuthology.orchestra.run.smithi058.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 9 -3051us[-3051us] +/- 102ms 2021-11-20T05:39:42.934 INFO:teuthology.orchestra.run.smithi058.stdout:^* hv02.front.sepia.ceph.com 2 6 377 75 +105us[ +82us] +/- 55ms 2021-11-20T05:39:42.934 INFO:teuthology.orchestra.run.smithi058.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +135us[ +135us] +/- 64ms 2021-11-20T05:39:42.934 INFO:teuthology.orchestra.run.smithi058.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 8 +1568us[+1568us] +/- 66ms 2021-11-20T05:39:42.936 INFO:teuthology.orchestra.run.smithi187.stdout:210 Number of sources = 4 2021-11-20T05:39:42.936 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-20T05:39:42.936 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2021-11-20T05:39:42.937 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 8 -3027us[-3019us] +/- 102ms 2021-11-20T05:39:42.937 INFO:teuthology.orchestra.run.smithi187.stdout:^* hv02.front.sepia.ceph.com 2 6 377 7 +132us[ +140us] +/- 56ms 2021-11-20T05:39:42.937 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +133us[ +141us] +/- 64ms 2021-11-20T05:39:42.938 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 10 +1551us[+1560us] +/- 66ms 2021-11-20T05:39:42.938 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-11-20T05:39:42.951 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2021-11-20T05:39:42.952 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2021-11-20T05:39:42.976 DEBUG:teuthology.orchestra.run.smithi058:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-11-20T05:39:43.042 DEBUG:teuthology.orchestra.run.smithi187:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-11-20T05:39:43.103 DEBUG:teuthology.orchestra.run.smithi058:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-20T05:39:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T05:39:43.133 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-20T05:39:43.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T05:39:43.161 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-11-20T05:39:43.177 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2021-11-20T05:39:43.203 INFO:teuthology.task.internal:Duration was 1244.459668 seconds 2021-11-20T05:39:43.204 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-11-20T05:39:43.236 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2021-11-20T05:39:43.237 DEBUG:teuthology.orchestra.run.smithi058:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-11-20T05:39:43.239 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-11-20T05:39:43.283 INFO:teuthology.orchestra.run.smithi058.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-20T05:39:43.284 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-20T05:39:43.477 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2021-11-20T05:39:43.478 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi058.front.sepia.ceph.com 2021-11-20T05:39:43.478 DEBUG:teuthology.orchestra.run.smithi058:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-11-20T05:39:43.500 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi187.front.sepia.ceph.com 2021-11-20T05:39:43.502 DEBUG:teuthology.orchestra.run.smithi187:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-11-20T05:39:43.523 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2021-11-20T05:39:43.524 DEBUG:teuthology.orchestra.run.smithi058:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-20T05:39:43.542 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-20T05:39:43.617 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2021-11-20T05:39:43.618 DEBUG:teuthology.orchestra.run.smithi058:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2021-11-20T05:39:43.622 DEBUG:teuthology.orchestra.run.smithi187:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2021-11-20T05:39:43.776 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2021-11-20T05:39:43.796 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2021-11-20T05:39:43.797 DEBUG:teuthology.orchestra.run.smithi058:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-11-20T05:39:43.823 DEBUG:teuthology.orchestra.run.smithi187:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-11-20T05:39:43.850 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2021-11-20T05:39:43.863 DEBUG:teuthology.orchestra.run.smithi058:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-11-20T05:39:43.868 DEBUG:teuthology.orchestra.run.smithi187:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-11-20T05:39:43.889 INFO:teuthology.orchestra.run.smithi058.stdout:kernel.core_pattern = core 2021-11-20T05:39:43.915 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = core 2021-11-20T05:39:43.928 DEBUG:teuthology.orchestra.run.smithi058:> test -e /home/ubuntu/cephtest/archive/coredump 2021-11-20T05:39:43.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T05:39:43.959 DEBUG:teuthology.orchestra.run.smithi187:> test -e /home/ubuntu/cephtest/archive/coredump 2021-11-20T05:39:43.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T05:39:43.984 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-11-20T05:39:43.999 INFO:teuthology.task.internal:Transferring archived files... 2021-11-20T05:39:44.000 DEBUG:teuthology.misc:Transferring archived files from smithi058:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514741/remote/smithi058 2021-11-20T05:39:44.001 DEBUG:teuthology.orchestra.run.smithi058:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-11-20T05:39:44.056 DEBUG:teuthology.misc:Transferring archived files from smithi187:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514741/remote/smithi187 2021-11-20T05:39:44.058 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-11-20T05:39:44.109 INFO:teuthology.task.internal:Removing archive directory... 2021-11-20T05:39:44.110 DEBUG:teuthology.orchestra.run.smithi058:> rm -rf -- /home/ubuntu/cephtest/archive 2021-11-20T05:39:44.112 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf -- /home/ubuntu/cephtest/archive 2021-11-20T05:39:44.161 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2021-11-20T05:39:44.180 INFO:teuthology.task.internal:Not uploading archives. 2021-11-20T05:39:44.180 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2021-11-20T05:39:44.203 INFO:teuthology.task.internal:Tidying up after the test... 2021-11-20T05:39:44.203 DEBUG:teuthology.orchestra.run.smithi058:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-11-20T05:39:44.206 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-11-20T05:39:44.221 INFO:teuthology.orchestra.run.smithi187.stdout: 262155 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 20 05:39 /home/ubuntu/cephtest 2021-11-20T15:34:08.231 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-11-20T15:34:08.263 DEBUG:teuthology.task.console_log:Killing console logger for smithi058 2021-11-20T15:34:08.264 DEBUG:teuthology.task.console_log:Killing console logger for smithi187 2021-11-20T15:34:08.265 DEBUG:teuthology.exit:Finished running handlers